[Patches] [Patch #103028] Make tempfile.mktemp threadsafe

[noreply@sourceforge.net]

Patch #103028 has been updated. 

Project: python
Category: library
Status: Open
Submitted by: nobody
Assigned to : gvanrossum
Summary: Make tempfile.mktemp threadsafe

Follow-Ups:

Date: 2001-Jan-11 01:08
By: tim_one

Comment:
Guido, I'm briefly assigning this to you just to get an answer to a
question:  if threads aren't supported on some platform, do we then get an
ImportError upon an attempt to do

import thread

?  If not, then how to check?  if "thread" in sys.builtin_module_names?

BTW, Tres's test case has surprising behavior on Windows (he ran it on
Linux):  it reports no errors, but that's because all of them pop up in the
os.unlink, after some other thread has already deleted the (same-named!)
temp file used by this thread.  Then the thread dies because the exception
is unhandled.  So there's a large burst of errors when the test starts, but
that kills off an equally large number of threads, so the test *appears*
<wink> to get better-behaved over time.

Note too that while the Summary line complains about mktemp, the Linux
failures are really in TemporaryFile:  mktemp returns the same name there
sometimes, and there's clearly a race between that and os.open() creating a
file of that (shared) name (i.e., mktemp's os.path.exists() test is
ineffective in such cases, and os.open() goes on to try to create that file
multiple times in O_EXCL mode).

I can't fix this without strong exclusion (i.e., a lock) -- or a collection
of platform-specific OS services for generating guaranteed-unique temp
files (e.g., on Windows there's

FILE* tmpfile(void);

which returns a unique stream opened in binary update mode).  The latter is
a lot more work than slopping in a lock, though (e.g., Windows tmpfile
attempts to create a file in the current working directory, & that's got
problems of its own).

-------------------------------------------------------

Date: 2001-Jan-02 09:02
By: tseaver

Comment:
Here is a testcase for the bug:

import threading, StringIO
from time import sleep
from traceback import print_exc

startEvent = threading.Event()
quitEvent = threading.Event()

SLEEP_INTERVAL = 0.2

class TempFileGreedy( threading.Thread ):
    """
    """
    error_count = 0
    ok_count = 0

    def run( self ):
        """
        """
        self.errors = StringIO.StringIO()
        startEvent.wait()

        while not quitEvent.isSet():
            try:
                f = tempfile.TemporaryFile( "w+b" )
            except:
                self.error_count = self.error_count + 1
                print_exc( file=self.errors )
            else:
                f.close()
                self.ok_count = self.ok_count + 1
            sleep( SLEEP_INTERVAL )

if __name__ == '__main__':

    import sys
    if len( sys.argv ) > 1:
        sys.path.insert( 0, '.' )

    import tempfile

    tempfile.gettempdir() # Do this now, to avoid spurious races later

    NUM_THREADS = 100
    threads = []

    print "Creating"
    for i in range( NUM_THREADS ):
        t = TempFileGreedy()
        threads.append( t )
        t.start()

    RUN_INTERVAL = 200.0

    print "Starting"
    startEvent.set()
    sleep( RUN_INTERVAL )
    quitEvent.set()

    print "Reaping"
    ok = errors = 0
    for thread in threads:
        thread.join()
        ok = ok + thread.ok_count
        errors = errors + thread.error_count
        if thread.error_count:
                print '%s errors:\n%s' % ( thread.getName()
                                         , thread.errors.getvalue()
                                         )

    print "Done"
    print 'OK:   %d\nErrors: %d' % ( ok, errors )

-------------------------------------------------------

Date: 2000-Dec-28 06:52
By: gvanrossum

Comment:
Randomly assigned to Tim.  I'd like mktemp to be thread-safe but I wish it
wouldn't have to use a lock. Is there any other way?

-------------------------------------------------------

Date: 2000-Dec-27 17:19
By: tseaver

Comment:
Doesn't need the 'global counter' in mktemp:

[/usr/lib/python1.5] $ diff -u tempfile.py.org tempfile.py
--- tempfile.py.org     Wed Dec 27 19:44:29 2000
+++ tempfile.py Wed Dec 27 20:05:57 2000
@@ -84,15 +84,35 @@
 
 counter = 0
 
+#
+#   In threaded Pythons, make this operation threadsafe.
+#
+try:
+    from threading import Lock
+    counterLock = Lock()
+except:
+    def _bumpCounter():
+        global counter
+        counter = counter + 1
+        return counter
+else:
+    def _bumpCounter():
+        global counter, counterLock
+        counterLock.acquire()
+        try:
+            counter = counter + 1
+            return counter
+        finally:
+            counterLock.release()
+
 
 # User-callable function to return a unique temporary file name
 
 def mktemp(suffix=""):
-    global counter
     dir = gettempdir()
     pre = gettempprefix()
     while 1:
-        counter = counter + 1
+        counter = _bumpCounter()
         file = os.path.join(dir, pre + `counter` + suffix)
         if not os.path.exists(file):
             return file

-------------------------------------------------------

-------------------------------------------------------
For more info, visit:

http://sourceforge.net/patch/?func=detailpatch&patch_id=103028&group_id=5470