[New-bugs-announce] [issue24303] OSError 17 due to _multiprocessing/semaphore.c assuming a one-to-one Pid -> process mapping.

[Paul Hobbs]

New submission from Paul Hobbs:

Using pid namespacing it is possible to have multiple processes with the same pid.  "semlock_new" creates a semaphore file with the template "/dev/shm/mp{pid}-{counter}".  This can conflict if the same semaphore file already exists due to another Python process have the same pid.

This bug has been fixed in Python3: https://bugs.python.org/issue8713.  However, that patch is very large (40 files, ~4.4k changed lines) and only incidentally fixes this bug while introducing a large backwards-incompatible refactoring and feature addition.

The following small patch to just _multiprocessing/semaphore.c fixes the problem by using the system clock and retrying to avoid conflicts:

--- a/Modules/_multiprocessing/semaphore.c
+++ b/Modules/_multiprocessing/semaphore.c
@@ -7,6 +7,7 @@
  */
 
 #include "multiprocessing.h"
+#include <time.h>
 
 enum { RECURSIVE_MUTEX, SEMAPHORE };
 
@@ -419,7 +420,7 @@ semlock_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
 {
     char buffer[256];
     SEM_HANDLE handle = SEM_FAILED;
-    int kind, maxvalue, value;
+    int kind, maxvalue, value, try;
     PyObject *result;
     static char *kwlist[] = {"kind", "value", "maxvalue", NULL};
     static int counter = 0;
@@ -433,10 +434,24 @@ semlock_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
         return NULL;
     }
 
-    PyOS_snprintf(buffer, sizeof(buffer), "/mp%ld-%d", (long)getpid(), counter++);
+    /* With pid namespaces, we may have multiple processes with the same pid.
+     * Instead of relying on the pid to be unique, we use the microseconds time
+     * to attempt to a unique filename. */
+    for (try = 0; try < 100; ++try) {
+        struct timespec tv;
+        long arbitrary = clock_gettime(CLOCK_REALTIME, &tv) ? 0 : tv.tv_nsec;
+        PyOS_snprintf(buffer, sizeof(buffer), "/mp%ld-%d-%ld",
+                      (long)getpid(),
+                      counter++,
+                      arbitrary);
+        SEM_CLEAR_ERROR();
+        handle = SEM_CREATE(buffer, value, maxvalue);
+        if (handle != SEM_FAILED)
+            break;
+        else if (errno != EEXIST)
+            goto failure;
+    }
 
-    SEM_CLEAR_ERROR();
-    handle = SEM_CREATE(buffer, value, maxvalue);
     /* On Windows we should fail if GetLastError()==ERROR_ALREADY_EXISTS */
     if (handle == SEM_FAILED || SEM_GET_LAST_ERROR() != 0)
         goto failure;

----------
messages: 244211
nosy: Paul Hobbs
priority: normal
severity: normal
status: open
title: OSError 17 due to _multiprocessing/semaphore.c assuming a one-to-one Pid -> process mapping.
type: enhancement
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue24303>
_______________________________________