On Tue, Aug 16, 2011 at 2:26 PM, Benjamin Peterson wrote:

David Townshend writes:

...

This idea was proposed in the discussion on shutil.move, but I thought it

would be worth posting separately to avoid confusing the discussion.

...

The idea is to add a create mode ('c') to the builtin open() function, which

will have the same effect as os.open(file, os.O_EXCL|os.O_CREAT).  I have added an issue (http://bugs.python.org/issue12760) for this, including a patch.

I am -1 because

- Possibly not portable or at least subject to implementations of varying quality. - No precedence in other languages or fopen() for that matter. - It's not hard to use os.fdopen().

Agreed. Also I think that in most cases the right thing to do is to quietly overwrite the file. If you're implementing a UI where you want look-before-you-leap, the app should code an explicit test so it can issue a proper error message (the exception will not be fun for the user :-). -- --Guido van Rossum (python.org/~guido)

On Tue, Aug 16, 2011 at 4:18 PM, Bruce Leban wrote:

On Tue, Aug 16, 2011 at 3:43 PM, Guido van Rossum wrote:

...

Agreed. Also I think that in most cases the right thing to do is to quietly overwrite the file. If you're implementing a UI where you want look-before-you-leap, the app should code an explicit test so it can issue a proper error message (the exception will not be fun for the user :-).

This isn't look before you leap. It's catching a failure when the file can't be created, just as you would if you don't have permission to write a file in that directory or the file already exists and is locked so you don't have permission to overwrite it. It just adds one more reason the file can't be written.

So what's the use case? In general when using a command line environment overwriting the file is what you *want* to happen. Like with Unix "foo >bar". I don't even think there *is* a shell syntax for not overwriting an existing file, though you can use >> to append instead of overwrite -- this is open(filename, 'a').

The app should already have code to translate those exceptions into human-readable error messages so I don't think that's a good objection. (If the implementation of this function in some OS needs LBYL then I think that's an unfortunate defect in those OS.)

Agreed.

I've had enough working with programs that do things like silently eat exceptions and I consider silently overwriting a file in the same class.

Always? How would you update an existing file if you can't overwrite files? -- --Guido van Rossum (python.org/~guido)

On Tue, Aug 16, 2011 at 5:12 PM, Bruce Leban wrote:

On Tue, Aug 16, 2011 at 4:46 PM, Guido van Rossum wrote:

...

So what's the use case? In general when using a command line environment overwriting the file is what you *want* to happen. Like with Unix "foo >bar". I don't even think there *is* a shell syntax for not overwriting an existing file, though you can use >> to append instead of overwrite -- this is open(filename, 'a').

We weren't just discussing command line tools.

Well if anything, Python is *lower* level than command line tools, not higher.

That said, I'm sure I'm not the only person on this list who has inadvertently overwritten a file using foo > bar. While we may have grown accustomed to this behavior and some people (like you) may even consider it desirable, not everyone does.

But are they right?

...

...

I've had enough working with programs that do things like silently eat exceptions and I consider silently overwriting a file in the same class.

Always? How would you update an existing file if you can't overwrite files?

I didn't say never overwrite. What I don't like is programs overwriting files without explicitly intending to do that.

Ah, but when you write open(fn, 'w') you *do* explicitly intend to overwrite. That's what it does.

Yes, there's a long legacy of overwriting files without warning or intent. I suppose I'm fighting an uphill battle (and it's not my highest priority complaint about bad code for that matter).

Right. -- --Guido van Rossum (python.org/~guido)

Guido van Rossum writes:

So what's the use case?

I agree that I don't see the use case that isn't already adequately covered. You generally get what you explicitly ask for, and that's how it should be. A point of correction, though:

In general when using a command line environment overwriting the file is what you *want* to happen. Like with Unix "foo >bar". I don't even think there *is* a shell syntax for not overwriting an existing file,

The ‘bash(1)’ man page describes the ‘noclobber’ option: If the redirection operator is >, and the noclobber option to the set builtin has been enabled, the redirection will fail if the file whose name results from the expansion of word exists and is a regular file. -- \ “If you go flying back through time and you see somebody else | `\ flying forward into the future, it's probably best to avoid eye | _o__) contact.” —Jack Handey | Ben Finney

Agreed. Also I think that in most cases the right thing to do is to quietly overwrite the file. If you're implementing a UI where you want look-before-you-leap, the app should code an explicit test so it can issue a proper error message (the exception will not be fun for the user :-).

The explicit test being "if not os.path.exists(pth)"? That has a race condition. LBYL is impossible here without a race condition, in fact, because the situation can change between looking and leaping. An exception, or else return code, is the only way. These can be checked for after the fact. I'd also point out that for those that don't want race conditions, Python is discouraging. The correct incantation involves two undocumented constants, plus a unique and rarely used way of opening files that involves unix file descriptors. Devin On Tue, Aug 16, 2011 at 6:43 PM, Guido van Rossum wrote:

On Tue, Aug 16, 2011 at 2:26 PM, Benjamin Peterson wrote:

...

David Townshend writes:

...

This idea was proposed in the discussion on shutil.move, but I thought it

would be worth posting separately to avoid confusing the discussion.

...

The idea is to add a create mode ('c') to the builtin open() function, which

will have the same effect as os.open(file, os.O_EXCL|os.O_CREAT).  I have added an issue (http://bugs.python.org/issue12760) for this, including a patch.

I am -1 because

- Possibly not portable or at least subject to implementations of varying quality. - No precedence in other languages or fopen() for that matter. - It's not hard to use os.fdopen().

Agreed. Also I think that in most cases the right thing to do is to quietly overwrite the file. If you're implementing a UI where you want look-before-you-leap, the app should code an explicit test so it can issue a proper error message (the exception will not be fun for the user :-).

-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Tue, Aug 16, 2011 at 11:49 PM, Benjamin Peterson wrote:

Devin Jeanpierre writes:

...

LBYL is impossible here without a race condition, in fact, because the situation can change between looking and leaping. An exception, or else return code, is the only way. These can be checked for after the fact.

How often in this used? In every application I've written, writing a file usually results from the user giving a path, in which case it's intended to replace whatever is already there.

Unless it isn't. Most GUI apps ask you to confirm whether you want to open a file even where one exists. Suppose you do a LBYL approach: you check to see if no file is there, then somebody writes a file there, then you overwrite it because you used 'w' mode because no file was there. It's not a disaster, since this is kind of hard to do by accident, but it is incorrect behavior if you wanted to actually ask if overwriting was kosher. On the other hand, if you ask _after_ trying to open the file, then one of two things can happen: the user says "abort", or the user says, "overwrite it". In the former case, we start over. In the latter case, the only remaining race condition is one that doesn't matter: the file might disappear before you overwrite it!

...

I'd also point out that for those that don't want race conditions, Python is discouraging. The correct incantation involves two undocumented constants, plus a unique and rarely used way of opening files that involves unix file descriptors.

If you truly want to avoid all filesystem race conditions, you're going to be dealing with file descriptors and low-level syscalls galore. Moving one aspect to a higher level is not too helpful on the whole.

Well, eh, not really. As far as I know this particular primitive is the probably the most important one. It's certainly the only one I've ever wanted to use Devin

Benjamin Peterson wrote:

Devin Jeanpierre writes:

...

LBYL is impossible here without a race condition, in fact, because the situation can change between looking and leaping. An exception, or else return code, is the only way. These can be checked for after the fact.

How often in this used? In every application I've written, writing a file usually results from the user giving a path, in which case it's intended to replace whatever is already there.

I frequently write scripts that, e.g. rename or move files with little or no human intervention. wget is a prime example of an application which needs to avoid overwriting files without human intervention, although I don't know how well it deals with race conditions. -- Steven

Perhaps we need a HOW-TO on working with files that discusses special-case needs and solutions that use os, tempfile, etc alternatives to builtin open.

Not a bad idea, but would it not get too complicated? Starting with ways to use os.open is fine, but to do it properly we would need to go on to copying and moving, and this would involve basically rewriting the implementation of shutils into the HOW-TO. I think that this is a good idea, but not enough on its own. My suggestion, to do this properly, would be to first implement the open create mode in open. Then update (or write new) copy functions (i.e. copyfile, copy2, copytree) to use the create mode, thereby effectively changing the linux shell commands represented from "cp" to "cp -n". Finally implement a new move function which, if possible, uses link/unlink or the immutable attribute (as discussed in the tread on shutil.move), and falls back to copy/unlink using the new copytree. The resulting functions would correspond to the "no-clobber" versions of the equivalent shell commands. The only problem is that O_EXCL may not be supported on all platforms. Can anyone tell me which platforms these are? I would like to see if I can find a way to achieve the same effect on those platforms, but so far I haven't been able to find out what they are. A HOW-TO would be useful to discuss other methods, such as tempfile, which will be a lot easier to use with the no-clobber versions. David

So if it is cross-platform, then what's the problem? On Thu, Aug 18, 2011 at 11:35 AM, Antoine Pitrou wrote:

On Thu, 18 Aug 2011 07:54:32 +0200 David Townshend wrote:

...

The only problem is that O_EXCL may not be supported on all platforms.

Can

...

anyone tell me which platforms these are?

That sounds unlikely. O_EXCL is a POSIX standard. It is also supported under Windows by the _open/_wopen compatibility functions (which we use for file I/O).

Probably there are very old systems which don't support it, and perhaps new systems that don't implement it *correctly* (meaning not atomically); for the former I'd say we just don't care (who's gonna run Python 3 on a 1995 system?) and for the latter, well, if the OS designers think it's fine, let's just expose it as it is.

As for NFS, there's an interesting comment from 2007 here: http://lwn.net/Articles/251971/

“My NFS tester shows that it at least appears to work with Linux, Solaris and FreeBSD: http://www.dovecot.org/list/dovecot/2007-July/024102.html. Looking at Linux 2.6 sources it doesn't look like it tries to implement a racy O_EXCL check in client side (fs/nfs/nfs3proc.c nfs3_proc_create()), so the test's results should be correct. I don't know if other OSes do that. I guess it would be nice to have a better O_EXCL tester which tries to catch race conditions.”

Regards

Antoine.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Wed, 17 Aug 2011 14:11:32 +1000 Nick Coghlan wrote:

On Wed, Aug 17, 2011 at 1:43 PM, Devin Jeanpierre wrote:

...

I'd also point out that for those that don't want race conditions, Python is discouraging. The correct incantation involves two undocumented constants, plus a unique and rarely used way of opening files that involves unix file descriptors.

FWIW, when you control the filename, you can include an additional subdirectory precisely for the exception when a second process attempts to create the same subdirectory.

How do you create a directory and a file atomically? Regards Antoine.

Le mercredi 17 août 2011 à 11:52 +0200, M.-A. Lemburg a écrit :

Antoine Pitrou wrote:

...

On Wed, 17 Aug 2011 14:11:32 +1000 Nick Coghlan wrote:

...

On Wed, Aug 17, 2011 at 1:43 PM, Devin Jeanpierre wrote:

...

I'd also point out that for those that don't want race conditions, Python is discouraging. The correct incantation involves two undocumented constants, plus a unique and rarely used way of opening files that involves unix file descriptors.

FWIW, when you control the filename, you can include an additional subdirectory precisely for the exception when a second process attempts to create the same subdirectory.

How do you create a directory and a file atomically?

On Windows, directories are created atomically. On Unix, too, but symlinks are faster. You can use those to implement cooperative file locks in a fairly cross-platform way.

I was thinking of creating both the directory and the file in a single atomic operation. But if the directory is only ever used for that file, I guess it's ok. (there's still a problem when deleting the directory and the file, which can't be atomic, and the file has to be deleted before the directory, meaning if the process crashes in between, there are "legitimate" situations where the directory exists but not the file in it...) Regards Antoine.

On Wed, 17 Aug 2011 12:17:04 +0200 "M.-A. Lemburg" wrote:

Antoine Pitrou wrote:

...

Le mercredi 17 août 2011 à 11:52 +0200, M.-A. Lemburg a écrit :

...

Antoine Pitrou wrote:

...

On Wed, 17 Aug 2011 14:11:32 +1000 Nick Coghlan wrote:

...

On Wed, Aug 17, 2011 at 1:43 PM, Devin Jeanpierre wrote:

...

I'd also point out that for those that don't want race conditions, Python is discouraging. The correct incantation involves two undocumented constants, plus a unique and rarely used way of opening files that involves unix file descriptors.

FWIW, when you control the filename, you can include an additional subdirectory precisely for the exception when a second process attempts to create the same subdirectory.

How do you create a directory and a file atomically?

On Windows, directories are created atomically. On Unix, too, but symlinks are faster. You can use those to implement cooperative file locks in a fairly cross-platform way.

I was thinking of creating both the directory and the file in a single atomic operation. But if the directory is only ever used for that file, I guess it's ok. (there's still a problem when deleting the directory and the file, which can't be atomic, and the file has to be deleted before the directory, meaning if the process crashes in between, there are "legitimate" situations where the directory exists but not the file in it...)

The directory is only used as locking mechanism. You normally don't need to create any files within that lock directory unless you want to store extra lock information.

Ah, I thought Nick proposed to create the file in that directory. My bad. Regards Antoine.

On Wed, 17 Aug 2011 13:42:59 +1000 Nick Coghlan wrote:

On Wed, Aug 17, 2011 at 12:10 PM, Calvin Spealman wrote:

...

While there are plenty of responders debating how unix-y or sensible this is from the standpoint of other languages or precedents, I think this makes sense from the standpoint of new users and a fresh viewpoints. Creating a new file is a very sensible thing to want in a single operation.

Yes, this suggestion came out of the shutil.move discussion, precisely *because* it is so hard to write a platform-independent, multi-process safe operation that will reliably either create a new file or else throw an exception if the file already exists,

Platform-independent is not hard. O_EXCL is specified by POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html Works under Linux:

...

...

import os os.open("LICENSE", os.O_CREAT | os.O_EXCL) Traceback (most recent call last): File "<stdin>", line 1, in <module> FileExistsError: [Errno 17] File exists: 'LICENSE'

Works under Windows:

...

...

import os os.open("LICENSE", os.O_CREAT | os.O_EXCL) Traceback (most recent call last): File "<stdin>", line 1, in <module> FileExistsError: [Errno 17] File exists

(yes, I'm taking the opportunity to showcase PEP 3151) Regards Antoine.