[Python-ideas] Implementation of shutil.move

[Calvin Spealman]

On Fri, Aug 12, 2011 at 9:20 AM, Mike Graham <mikegraham at gmail.com> wrote:
> On Fri, Aug 12, 2011 at 7:59 AM, David Townshend <aquavitae69 at gmail.com> wrote:
>> The shutil.move function uses os.rename to move files on the same file
>> system. On unix, this function will overwrite an existing destination, so
>> the obvious approach is
>> if not os.path.exists(dst):
>>     shutil.move(src, dst)
>> But this could result in race conditions if dst is created after
>> os.path.exists and before shutil.move.  From my research, it seems that this
>> is a limitation in the unix c library, but it should be possible to avoid it
>> through a workaround (pieced together
>> from http://bytes.com/topic/python/answers/555794-safely-renaming-file-without-overwriting).
>>  This involves some fairly low-level work, so I propose adding a new move2
>> function to shutil, which raises an error if dst exists and locking it if it
>> doesn't:
>> def move2(src, dst):
>>     try:
>>         fd = os.open(dst, os.O_EXCL | os.O_CREAT)
>>     except OSError:
>>         raise Error('Destination exists')
>>     try:
>>         move(src, dst)
>>     finally:
>>         os.close(fd)
>> This could be optimised by using shutil.move code rather than just calling
>> it, but the idea is that an attempt is made to create dst with exclusive
>> access. If this fails, then it means that the file exists, but if it passes,
>> then dst is locked so no other process can create it.
>
> This type of problem comes up regularly and a lot of user code is
> riddled with this kind of race conditions. Many (most?) are avoidable
> now by writing code to using EAFP rather than LBYL, but many are not.
> I wonder if this broader problem could be addressed by a context
> manager.
>
> Something to the general effect of the usage
>
> try:
>    with lockfile(dst):
>        move(src, dst)
> except OSError as e:
>    if e != errno.EEXIST:
>        raise
>    raise AppSpecificError("File already exists.") # or whatever
>
> and a definition like
>
> @contextlib.contextmanager
> def lockfile(path):
>    fd = os.open(path, os.O_EXCL | os.O_CREAT)
>    yield
>    os.close(fd)
>
>
> The usage is still sort of ugly, but I'm not sure I can think of a
> general way that isn't.

lockfile is not a good name for this, but the manager itself is very
simple and useful. +1 from me for getting this somewhere into stdlib.

> Mike
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy