[ python-Bugs-1223238 ] race in os.makedirs()

[SourceForge.net]

Bugs item #1223238, was opened at 2005-06-18 16:37
Message generated for change (Comment added) made by gbrandl
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1223238&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Mattias Engdegård (yorick)
Assigned to: Nobody/Anonymous (nobody)
Summary: race in os.makedirs()

Initial Comment:
os.makedirs() can fail if one of its components is
created while it is running (perhaps by another
process). This is because it checks for each directory
if it exists before creating it.

This is bad programming style. A correct implementation
would just call mkdir() on each directory (starting
with the rightmost, probably) and ignoring any EEXIST
error. This would not only fix the bug, it would also
be faster (fewer syscalls).

The patch is simple, but there is a wart in the design:
os.makedirs() throws an error if the (rightmost)
directory already exists, although by calling this
function the user has clearly indicated that she wants
the directories to be created if they don't exist and
have no complaints otherwise.

This leads to code like:

try:
    os.makedirs(path)
except OSError:
    pass

which is doubly bad because it hides the race condition!

So, before I submit a patch, should we:
a) just fix this bug but keep the old design
b) fix this bug, and don't throw an error if the dir exists

or maybe do a) for the next 2.4.x bugfix release and b)
in 2.5?


----------------------------------------------------------------------

>Comment By: Georg Brandl (gbrandl)
Date: 2006-12-09 09:25

Message:
Logged In: YES 
user_id=849994
Originator: NO

Fixed by patch #1608267.

----------------------------------------------------------------------

Comment By: Nir Soffer (nirs)
Date: 2005-12-05 01:32

Message:
Logged In: YES 
user_id=832344

I agree that raising an error for existing directories is usually not what
you 
want, but changing this will break any code that count on that documented

behavior.


----------------------------------------------------------------------

Comment By: Mattias Engdegård (yorick)
Date: 2005-07-18 12:08

Message:
Logged In: YES 
user_id=432579

Whether the dir creation is done right-to-left or
left-to-right is less important. If the expected usage
pattern is that most of the directories already exist, then
right-to-left may be faster, otherwise left-to-right is. One
advantage with the former is its slightly simpler code (no
need to check for ENOENT).

>current 2.4 code does not return an error if the directory
exists,
>the patch must not change that behavior.

You mean the contrary? From what I can see of the 2.4 code,
it throws an error if the directory exists. This is almost
never what you want, so I strongly doubt fixing that
misfeature in 2.5 would break anything. I'm happy with the
suggested patch for 2.5 in #1239890.


----------------------------------------------------------------------

Comment By: Nir Soffer (nirs)
Date: 2005-07-17 21:10

Message:
Logged In: YES 
user_id=832344

current 2.4 code does not return an error if the directory exists, the
patch 
must not change that behavior.

It will not be a good idea to change that behavior in 2.5 or any version,
it 
can break lot of code.

----------------------------------------------------------------------

Comment By: Georg Brandl (birkenfeld)
Date: 2005-07-17 19:56

Message:
Logged In: YES 
user_id=1188172

See patch #1239890.

----------------------------------------------------------------------

Comment By: Mattias Engdegård (yorick)
Date: 2005-06-25 21:11

Message:
Logged In: YES 
user_id=432579

I'm fine with fixing the design for 2.5 and ignoring the bug for 2.4,
since 
programs susceptible to the bug must use some kind of work-around in 
2.4.x (x < 2) anyway.
What I am using right now is:

def makedirs(name, mode=0777):
    try:
        os.mkdir(name, mode)
        return
    except OSError, err:
        if err.errno == errno.EEXIST:
            return
        if err.errno != errno.ENOENT:
            raise
    makedirs(os.path.dirname(name), mode)
    makedirs(name, mode)

This is compact and elegant, but relies on mkdir producing the correct 
errno values, which should be true for all platforms I'm aware of. It
could 
also theoretically loop infinitely in bizarre cases but I don't see how
that 
ever could happen.


----------------------------------------------------------------------

Comment By: Neil Schemenauer (nascheme)
Date: 2005-06-18 17:43

Message:
Logged In: YES 
user_id=35752

I vote to fix the design for 2.5.  Backporting the minimal
fix to 2.4 would be optional, IMO.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1223238&group_id=5470