[issue8604] Adding an atomic FS write API

[STINNER Victor]

STINNER Victor added the comment:

Here is a patch based on the new os.replace() function. I only tested the patch on Linux.

My patch must be tested on Windows, I don't know what happens when the file object is closed when the NamedTemporaryFile.delete attribute is set to False. Does close() raise an exception?

I hesitate between prefix and dir parameters for NamedTemporaryFile. If prefix is equal to filename, is it possible for NamedTemporaryFile to create a file called filename? Or does it always add a suffix? If the program crashs before rename, I prefer to leave "file.XXX" temporary files (ex: "file.tmp", "file.tmp.2", ...) instead of random names like "tmpXXX".

It is important to create the temporary file in the same directory to not create a file on a different file system. os.replace() fails on POSIX if the source and the destination are on two different file systems.

If importing tempfile in shutil is a problem, it may be done in atomic_write() (maybe using a global variable to ensure that the module is only imported once).

--

First, I tried to handle the temporary file manually (without the tempfile module) with a constant suffix. I only created a temporary file if the creation of the file (in exclusive mode, using "x" mode") failed.

But Antoine pointed me on IRC that the function is not atomic because the file may be invalid before the file content is completly written and flushed. If two threads try to create the same file, the second thread gets a FileExistsError. In this case, the caller has to handle FileExistsError. The caller may remove the temporary file of the first thread, which may make the situation even worse.

I prefer to use tempfile.NamedTemporaryFile because it is portable and well tested. It also uses a random suffix. On Windows, the O_TEMPORARY flag is passed to os.open(), so the file is removed by the OS when the file is closed, or when the program does crash.

----------
keywords: +patch
Added file: http://bugs.python.org/file30273/atomic_write.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8604>
_______________________________________