[Python-ideas] shutil.symlink to allow non-race replacement of existing link targets

Tom Hale tom at hale.ee
Mon May 13 05:38:08 EDT 2019


As suggested by Toshio Kuratomi at https://bugs.python.org/issue36656, I 
am raising this here for inclusion in the shutil module.

Mimicking POSIX, os.symlink() will raise FileExistsError if the link 
name to be created already exists.

A common use case is overwriting an existing file (often a symlink) with 
a symlink. Naively, one would delete the file named link_name file if it 
exists, then call symlink(). This "solution" is already 3 lines of code, 
and without exception handling it introduces the race condition of a 
file named link_name being created between unlink and symlink.

Depending on the functionality required, I suggest:

* os.symlink() - the new link name is expected to NOT exist
* shutil.symlink() - the new symlink replaces an existing file

Handling all possible race conditions (some detailed in issue36656) is 
non-trivial, however this is the best that I have come up with so far:

==========================================================================

import os, tempfile

def symlink(target, link_name):
     '''Create a symbolic link link_name pointing to target.
     Overwrites link_name if it exists. '''

     # os.replace() may fail if files are on different filesystems
     link_dir = os.path.dirname(link_name)

     # Link to a temporary filename that doesn't exist
     while True:
         temp_link_name = tempfile.mktemp(dir=link_dir)

         # os.* functions mimic as closely as possible system functions
         # The POSIX symlink() returns EEXIST if link_name already exists
         # 
https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html
         try:
             os.symlink(target, temp_link_name)
             break
         except FileExistsError:
             pass

     # Replace link_name with temp_link_name
     try:
         # Pre-empt os.replace on a directory with a nicer message
         if os.path.isdir(link_name):
             raise IsADirectoryError(f"Cannot symlink over existing 
directory: '{link_name}'")
         os.replace(temp_link_name, link_name)
     except:
         if os.path.islink(temp_link_name):
             os.remove(temp_link_name)
         raise

==========================================================================

The documentation (https://docs.python.org/3/library/shutil.html) I 
suggest for this is:

shutil.symlink(target, link_name)
Create a symbolic link named link_name pointing to target, overwriting 
target if it exists. If link_name is a directory, IsADirectoryError is 
raised. To not overwrite target, use os.symlink()

==========================================================================

It would be tempting to do:

while True:
     try:
         os.symlink(target, link_name)
         break
     except FileExistsError:
         os.remove(link_name)

But this has a race condition when replacing a symlink should should 
*always* exist, eg:

     /lib/critical.so -> /lib/critical.so.1.2

When upgrading by:

     symlink('/lib/critical.so.2.0', '/lib/critical.so')

There is a point in time when /lib/critical.so doesn't exist.

==========================================================================

One issue I see with my suggested code is that the file at 
temp_link_name could be changed before target is replaced with it. This 
is mitigated by the randomness introduced by mktemp().

While it is far less likely that a file is accessed with a random and 
unknown name than with an existing known name, I seek input on a 
solution if this is an unacceptable risk.


Prior art:

* https://bugs.python.org/issue36656 (already mentioned above)
* https://stackoverflow.com/a/55742015/5353461
* https://git.savannah.gnu.org/cgit/coreutils.git/tree/src/ln.c

-- 
Tom Hale


More information about the Python-ideas mailing list