[Python-ideas] Implementation of shutil.move

[Mike Graham]

On Fri, Aug 12, 2011 at 7:59 AM, David Townshend <aquavitae69 at gmail.com> wrote:
> The shutil.move function uses os.rename to move files on the same file
> system. On unix, this function will overwrite an existing destination, so
> the obvious approach is
> if not os.path.exists(dst):
>     shutil.move(src, dst)
> But this could result in race conditions if dst is created after
> os.path.exists and before shutil.move.  From my research, it seems that this
> is a limitation in the unix c library, but it should be possible to avoid it
> through a workaround (pieced together
> from http://bytes.com/topic/python/answers/555794-safely-renaming-file-without-overwriting).
>  This involves some fairly low-level work, so I propose adding a new move2
> function to shutil, which raises an error if dst exists and locking it if it
> doesn't:
> def move2(src, dst):
>     try:
>         fd = os.open(dst, os.O_EXCL | os.O_CREAT)
>     except OSError:
>         raise Error('Destination exists')
>     try:
>         move(src, dst)
>     finally:
>         os.close(fd)
> This could be optimised by using shutil.move code rather than just calling
> it, but the idea is that an attempt is made to create dst with exclusive
> access. If this fails, then it means that the file exists, but if it passes,
> then dst is locked so no other process can create it.

This type of problem comes up regularly and a lot of user code is
riddled with this kind of race conditions. Many (most?) are avoidable
now by writing code to using EAFP rather than LBYL, but many are not.
I wonder if this broader problem could be addressed by a context
manager.

Something to the general effect of the usage

try:
    with lockfile(dst):
        move(src, dst)
except OSError as e:
    if e != errno.EEXIST:
        raise
    raise AppSpecificError("File already exists.") # or whatever

and a definition like

@contextlib.contextmanager
def lockfile(path):
    fd = os.open(path, os.O_EXCL | os.O_CREAT)
    yield
    os.close(fd)


The usage is still sort of ugly, but I'm not sure I can think of a
general way that isn't.

Mike