What can I do without GIL?
Hello,
I a thread does not hold the GIL, is it totally forbidden to call any Py* API functions at all, or are there some exceptions?
More concretely, am I allowed to create fresh Python objects with PyBytes_FromString et. al. without holding the GIL?
Thanks,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
Nikolaus Rath, 18.10.2010 04:02:
I a thread does not hold the GIL, is it totally forbidden to call any Py* API functions at all, or are there some exceptions?
More concretely, am I allowed to create fresh Python objects with PyBytes_FromString et. al. without holding the GIL?
No. Anything that requires Python memory management and especially reference counting must be protected by the GIL. You can read the C pointer and length of a Python string without the GIL (using the C-API macros) and you can acquire and release thread locks, but you can't create new objects.
Even simple looking functions may raise exceptions in some corner cases (such as memory errors), which also requires the GIL to work. So don't expect there to be much that you can do without holding it. The best way to tell is to look through the CPython implementation of the function you want to call, including all other functions that it calls, and to check for anything that looks like reference counting or exception handling.
Stefan
On 10/18/2010 02:53 AM, Stefan Behnel wrote:
Nikolaus Rath, 18.10.2010 04:02:
I a thread does not hold the GIL, is it totally forbidden to call any Py* API functions at all, or are there some exceptions?
More concretely, am I allowed to create fresh Python objects with PyBytes_FromString et. al. without holding the GIL?
No. Anything that requires Python memory management and especially reference counting must be protected by the GIL. You can read the C pointer and length of a Python string without the GIL (using the C-API macros) and you can acquire and release thread locks, but you can't create new objects.
I see, thanks.
I'd like to follow up with another question directly: if I call PyBytes_AsString() on a Python object (and keep a reference to that object), can I rely on the char* to stay valid when I am releasing the GIL?
I just learned that in OCAML, the interpreter may happily move data around so that pointers are not guaranteed to stay valid if the GIL is released.
Is this a problem with Python as well, or can I rely on the string staying at the same memory location all the time?
Thanks,
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
Nikolaus Rath, 24.10.2010 18:09:
On 10/18/2010 02:53 AM, Stefan Behnel wrote:
Nikolaus Rath, 18.10.2010 04:02:
I a thread does not hold the GIL, is it totally forbidden to call any Py* API functions at all, or are there some exceptions?
More concretely, am I allowed to create fresh Python objects with PyBytes_FromString et. al. without holding the GIL?
No. Anything that requires Python memory management and especially reference counting must be protected by the GIL. You can read the C pointer and length of a Python string without the GIL (using the C-API macros) and you can acquire and release thread locks, but you can't create new objects.
I'd like to follow up with another question directly: if I call PyBytes_AsString() on a Python object (and keep a reference to that object), can I rely on the char* to stay valid when I am releasing the GIL?
First of all, you shouldn't call PyBytes_AsString() without holding the GIL because it may raise an exception. You can call PyBytes_AS_STRING(), though. As I said, check the implementation.
I just learned that in OCAML, the interpreter may happily move data around so that pointers are not guaranteed to stay valid if the GIL is released.
Is this a problem with Python as well, or can I rely on the string staying at the same memory location all the time?
CPython doesn't do that. Strings are immutable, so they never need to be reallocated by Python runtime operations. And CPython is tuned for making it easy to interact with C code through its C-API. One feature is that the memory management doesn't move stuff around that may be referenced by some C code somewhere.
Stefan
On 10/24/2010 12:57 PM, Stefan Behnel wrote:
Nikolaus Rath, 24.10.2010 18:09:
On 10/18/2010 02:53 AM, Stefan Behnel wrote:
Nikolaus Rath, 18.10.2010 04:02:
I a thread does not hold the GIL, is it totally forbidden to call any Py* API functions at all, or are there some exceptions?
More concretely, am I allowed to create fresh Python objects with PyBytes_FromString et. al. without holding the GIL?
No. Anything that requires Python memory management and especially reference counting must be protected by the GIL. You can read the C pointer and length of a Python string without the GIL (using the C-API macros) and you can acquire and release thread locks, but you can't create new objects.
I'd like to follow up with another question directly: if I call PyBytes_AsString() on a Python object (and keep a reference to that object), can I rely on the char* to stay valid when I am releasing the GIL?
First of all, you shouldn't call PyBytes_AsString() without holding the GIL because it may raise an exception. You can call PyBytes_AS_STRING(), though. As I said, check the implementation.
That's what I meant. I want to call PyBytes_* and *afterwards* release the GIL.
I just learned that in OCAML, the interpreter may happily move data around so that pointers are not guaranteed to stay valid if the GIL is released.
Is this a problem with Python as well, or can I rely on the string staying at the same memory location all the time?
CPython doesn't do that. Strings are immutable, so they never need to be reallocated by Python runtime operations. And CPython is tuned for making it easy to interact with C code through its C-API. One feature is that the memory management doesn't move stuff around that may be referenced by some C code somewhere.
Great, thanks again!
-Nikolaus
-- »Time flies like an arrow, fruit flies like a Banana.«
PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C
On Oct 24, 2010, at 9:09 AM, Nikolaus Rath <Nikolaus@rath.org> wrote:
I just learned that in OCAML, the interpreter may happily move data around so that pointers are not guaranteed to stay valid if the GIL is released.
Is this a problem with Python as well, or can I rely on the string staying at the same memory location all the time?
OCaml has a compacting garbage collector. Every once in a while, it will move everything around to reduce memory fragmentation. As far as I know, Python doesn't do this.
Nikolaus Rath <Nikolaus@rath.org> writes:
I'd like to follow up with another question directly: if I call PyBytes_AsString() on a Python object (and keep a reference to that object), can I rely on the char* to stay valid when I am releasing the GIL? [...] Is this a problem with Python as well, or can I rely on the string staying at the same memory location all the time?
CPython string data is stored inside the PyStringObject structure, so it is pretty much guaranteed to be immovable in the current implementation.
typedef struct { PyObject_VAR_HEAD long ob_shash; int ob_sstate; char ob_sval[1]; // PyString_AS_STRING returns pointer to this member } PyStringObject;
Nikolaus Rath <Nikolaus@rath.org> writes:
I a thread does not hold the GIL, is it totally forbidden to call any Py* API functions at all, or are there some exceptions?
More concretely, am I allowed to create fresh Python objects with PyBytes_FromString et. al. without holding the GIL?
You are not allowed to create fresh Python objects without holding the GIL. The objects might implement a caching strategy which would surely rely on the GIL for locking.
The only Python API calls that could conceivably be safe without holding the GIL are accessors to C members of immutable objects whose reference you're owning - say, PyString_AS_STRING, or PyInt_AS_LONG.
Nikolaus Rath, 18.10.2010 04:02:
I a thread does not hold the GIL, is it totally forbidden to call any Py* API functions at all, or are there some exceptions?
More concretely, am I allowed to create fresh Python objects with PyBytes_FromString et. al. without holding the GIL?
No. Anything that requires Python memory management and especially reference counting must be protected by the GIL. You can read the C pointer and length of a Python string without the GIL (using the C-API macros) and you can acquire and release thread locks, but you can't create new objects.
Even simple looking functions may raise exceptions in some corner cases (such as memory errors), which also requires the GIL to work. So don't expect there to be much that you can do without holding it. The best way to tell is to look through the CPython implementation of the function you want to call, including all other functions that it calls, and to check for anything that looks like reference counting or exception handling.
Stefan
participants (5)
-
Hrvoje Niksic -
Jason Baker -
Nikolaus Rath -
Stefan Behnel -
Stefan Behnel