Release or not release the GIL
Hi, While working on the implementation of the PEP 433, I saw different places where Python asks for operating resources without releasing the GIL. In my implementation, I released the GIL in the following places. Is it correct? - os.dup() - os.dup2() - os.pipe() - socket.socketpair() os.listdir() releases the GIL to call dup(), whereas os.dup() doesn't. Victor
I'm guessing those system calls are just implemented by the kernel and
cannot block for I/O, so it was deemed unnecessary to release the GIL
around them. I don't mind changing that though, you can never know
what happens when you make a system call.
On Thu, Jan 31, 2013 at 4:07 PM, Victor Stinner
Hi,
While working on the implementation of the PEP 433, I saw different places where Python asks for operating resources without releasing the GIL.
In my implementation, I released the GIL in the following places. Is it correct?
- os.dup() - os.dup2() - os.pipe() - socket.socketpair()
os.listdir() releases the GIL to call dup(), whereas os.dup() doesn't.
Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On 01/02/2013 1:44am, Guido van Rossum wrote:
I'm guessing those system calls are just implemented by the kernel and cannot block for I/O, so it was deemed unnecessary to release the GIL around them. I don't mind changing that though, you can never know what happens when you make a system call.
close() can block, so I assume the same is true of dup2(), since dup2(oldfd, newfd) closes oldfd. -- Richard
dup2(oldfd, newfd) closes oldfd.
No, it doesn't close oldfd.
It may close newfd if it was already open.
(I guess that's what he meant). Anyway, only dup2() should probably release the GIL. One reasonable heuristic is to check the man page: if the syscall can return EINTR, then the GIL should be released.
2013/2/1 Charles-François Natali
dup2(oldfd, newfd) closes oldfd.
No, it doesn't close oldfd.
It may close newfd if it was already open.
(I guess that's what he meant).
Anyway, only dup2() should probably release the GIL.
One reasonable heuristic is to check the man page: if the syscall can return EINTR, then the GIL should be released.
Should the call be retried in the EINTR case? (After a PyErr_CheckSignals) -- Amaury Forgeot d'Arc
Le Fri, 1 Feb 2013 15:18:39 +0100,
"Amaury Forgeot d'Arc"
2013/2/1 Charles-François Natali
dup2(oldfd, newfd) closes oldfd.
No, it doesn't close oldfd.
It may close newfd if it was already open.
(I guess that's what he meant).
Anyway, only dup2() should probably release the GIL.
One reasonable heuristic is to check the man page: if the syscall can return EINTR, then the GIL should be released.
Should the call be retried in the EINTR case? (After a PyErr_CheckSignals)
I don't think we want to retry low-level system calls (but I'm not sure we're currently consistent in that regard). Regards Antoine.
On Fri, Feb 1, 2013 at 7:22 AM, Antoine Pitrou
Le Fri, 1 Feb 2013 15:18:39 +0100, "Amaury Forgeot d'Arc"
a écrit : 2013/2/1 Charles-François Natali
dup2(oldfd, newfd) closes oldfd.
No, it doesn't close oldfd.
It may close newfd if it was already open.
(I guess that's what he meant).
Anyway, only dup2() should probably release the GIL.
One reasonable heuristic is to check the man page: if the syscall can return EINTR, then the GIL should be released.
Should the call be retried in the EINTR case? (After a PyErr_CheckSignals)
I don't think we want to retry low-level system calls (but I'm not sure we're currently consistent in that regard).
I think this is what you meant but to be clear: Anywhere we're using them within a library for a good purpose, we do need to retry. If we're merely exposing them via the os module such as os.dup, its up to the caller to deal with the retry.
On Fri, 1 Feb 2013 15:25:27 -0800
"Gregory P. Smith"
On Fri, Feb 1, 2013 at 7:22 AM, Antoine Pitrou
wrote: Le Fri, 1 Feb 2013 15:18:39 +0100, "Amaury Forgeot d'Arc"
a écrit : 2013/2/1 Charles-François Natali
dup2(oldfd, newfd) closes oldfd.
No, it doesn't close oldfd.
It may close newfd if it was already open.
(I guess that's what he meant).
Anyway, only dup2() should probably release the GIL.
One reasonable heuristic is to check the man page: if the syscall can return EINTR, then the GIL should be released.
Should the call be retried in the EINTR case? (After a PyErr_CheckSignals)
I don't think we want to retry low-level system calls (but I'm not sure we're currently consistent in that regard).
I think this is what you meant but to be clear: Anywhere we're using them within a library for a good purpose, we do need to retry. If we're merely exposing them via the os module such as os.dup, its up to the caller to deal with the retry.
Indeed, that's what I meant. Sorry for not being very clear. Regards Antoine.
Amaury Forgeot d'Arc wrote:
One reasonable heuristic is to check the man page: if the syscall can return EINTR, then the GIL should be released.
Hmmm, curious. According to the MacOSX man pages, both dup() and dup2() can return EINTR, but pipe() and socketpair() can't. I'm particularly surprised about dup(), as that's the one I would have thought couldn't possibly block -- it only needs to manipulate in-memory data structures. For Linux, the only one returning EINTR is dup2(). FreeBSD has none of them returning EINTR. So it looks like there's a fair bit of variation between unices here. -- Greg
participants (8)
-
Amaury Forgeot d'Arc
-
Antoine Pitrou
-
Charles-François Natali
-
Greg Ewing
-
Gregory P. Smith
-
Guido van Rossum
-
Richard Oudkerk
-
Victor Stinner