This idea was proposed in the discussion on shutil.move, but I thought it would be worth posting separately to avoid confusing the discussion. The idea is to add a create mode ('c') to the builtin open() function, which will have the same effect as os.open(file, os.O_EXCL|os.O_CREAT). I have added an issue (http://bugs.python.org/issue12760) for this, including a patch.
David Townshend <aquavitae69@...> writes:
This idea was proposed in the discussion on shutil.move, but I thought it
would be worth posting separately to avoid confusing the discussion.
The idea is to add a create mode ('c') to the builtin open() function, which
will have the same effect as os.open(file, os.O_EXCL|os.O_CREAT). I have added an issue (http://bugs.python.org/issue12760) for this, including a patch. I am -1 because - Possibly not portable or at least subject to implementations of varying quality. - No precedence in other languages or fopen() for that matter. - It's not hard to use os.fdopen().
On Tue, Aug 16, 2011 at 2:26 PM, Benjamin Peterson <benjamin@python.org> wrote:
Agreed. Also I think that in most cases the right thing to do is to quietly overwrite the file. If you're implementing a UI where you want look-before-you-leap, the app should code an explicit test so it can issue a proper error message (the exception will not be fun for the user :-). -- --Guido van Rossum (python.org/~guido)
On Tue, Aug 16, 2011 at 3:43 PM, Guido van Rossum <guido@python.org> wrote:
This isn't look before you leap. It's catching a failure when the file can't be created, just as you would if you don't have permission to write a file in that directory or the file already exists and is locked so you don't have permission to overwrite it. It just adds one more reason the file can't be written. The app should already have code to translate those exceptions into human-readable error messages so I don't think that's a good objection. (If the implementation of this function in some OS needs LBYL then I think that's an unfortunate defect in those OS.) I've had enough working with programs that do things like silently eat exceptions and I consider silently overwriting a file in the same class. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On Tue, Aug 16, 2011 at 4:18 PM, Bruce Leban <bruce@leapyear.org> wrote:
So what's the use case? In general when using a command line environment overwriting the file is what you *want* to happen. Like with Unix "foo >bar". I don't even think there *is* a shell syntax for not overwriting an existing file, though you can use >> to append instead of overwrite -- this is open(filename, 'a').
Agreed.
I've had enough working with programs that do things like silently eat exceptions and I consider silently overwriting a file in the same class.
Always? How would you update an existing file if you can't overwrite files? -- --Guido van Rossum (python.org/~guido)
On Tue, Aug 16, 2011 at 4:46 PM, Guido van Rossum <guido@python.org> wrote:
We weren't just discussing command line tools. That said, I'm sure I'm not the only person on this list who has inadvertently overwritten a file using foo > bar. While we may have grown accustomed to this behavior and some people (like you) may even consider it desirable, not everyone does.
I didn't say never overwrite. What I don't like is programs overwriting files without explicitly intending to do that. Yes, there's a long legacy of overwriting files without warning or intent. I suppose I'm fighting an uphill battle (and it's not my highest priority complaint about bad code for that matter). --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On Tue, Aug 16, 2011 at 5:12 PM, Bruce Leban <bruce@leapyear.org> wrote:
Well if anything, Python is *lower* level than command line tools, not higher.
But are they right?
Ah, but when you write open(fn, 'w') you *do* explicitly intend to overwrite. That's what it does.
Right. -- --Guido van Rossum (python.org/~guido)
Guido van Rossum wrote:
On Tue, Aug 16, 2011 at 5:12 PM, Bruce Leban <bruce@leapyear.org> wrote:
No. I rarely intend to over-write an existing file. Usually I intend to create a new file. So I nearly always do this: if not os.path.exists(fn): open(fn, 'w') in the full knowledge that there's a race condition there, and that if I have multiple processes writing to files at the same time, as I often do, I could very well lose data. And don't think for one second I'm even close to happy about that, but at least it is *some* protection, even if not very much.
I'm not sure that a "create" mode is the right solution, but I am sure that, just like Bruce, I want a good, battle-hardened, standard, platform independent (as much as possible) solution to the above race condition bug. Perhaps it should be a module, like the tempfile module. (There's a thread about adding something like this to shutil.) -- Steven
Guido van Rossum <guido@python.org> writes:
So what's the use case?
I agree that I don't see the use case that isn't already adequately covered. You generally get what you explicitly ask for, and that's how it should be. A point of correction, though:
The ‘bash(1)’ man page describes the ‘noclobber’ option: If the redirection operator is >, and the noclobber option to the set builtin has been enabled, the redirection will fail if the file whose name results from the expansion of word exists and is a regular file. -- \ “If you go flying back through time and you see somebody else | `\ flying forward into the future, it's probably best to avoid eye | _o__) contact.” —Jack Handey | Ben Finney
The explicit test being "if not os.path.exists(pth)"? That has a race condition. LBYL is impossible here without a race condition, in fact, because the situation can change between looking and leaping. An exception, or else return code, is the only way. These can be checked for after the fact. I'd also point out that for those that don't want race conditions, Python is discouraging. The correct incantation involves two undocumented constants, plus a unique and rarely used way of opening files that involves unix file descriptors. Devin On Tue, Aug 16, 2011 at 6:43 PM, Guido van Rossum <guido@python.org> wrote:
Devin Jeanpierre <jeanpierreda@...> writes:
How often in this used? In every application I've written, writing a file usually results from the user giving a path, in which case it's intended to replace whatever is already there.
If you truly want to avoid all filesystem race conditions, you're going to be dealing with file descriptors and low-level syscalls galore. Moving one aspect to a higher level is not too helpful on the whole.
On Tue, Aug 16, 2011 at 11:49 PM, Benjamin Peterson <benjamin@python.org> wrote:
Unless it isn't. Most GUI apps ask you to confirm whether you want to open a file even where one exists. Suppose you do a LBYL approach: you check to see if no file is there, then somebody writes a file there, then you overwrite it because you used 'w' mode because no file was there. It's not a disaster, since this is kind of hard to do by accident, but it is incorrect behavior if you wanted to actually ask if overwriting was kosher. On the other hand, if you ask _after_ trying to open the file, then one of two things can happen: the user says "abort", or the user says, "overwrite it". In the former case, we start over. In the latter case, the only remaining race condition is one that doesn't matter: the file might disappear before you overwrite it!
Well, eh, not really. As far as I know this particular primitive is the probably the most important one. It's certainly the only one I've ever wanted to use Devin
It seems that the only problem raised which hasn't been already responded to is that it won't necessarily work on all platforms. Could anyone elaborate on which platforms it won't work on? From what I've read it will work on all unix and windows systems which I think covers the majority. I can't see that it would be a problem if it doesn't work on a few specialised systems as long as the documentation is clear on this (any it might turn out that it can be handled, just through os-specific code). It at least allows the feature to be available for the 90% of use cases when it will work. David On Wed, Aug 17, 2011 at 6:13 AM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
Benjamin Peterson wrote:
I frequently write scripts that, e.g. rename or move files with little or no human intervention. wget is a prime example of an application which needs to avoid overwriting files without human intervention, although I don't know how well it deals with race conditions. -- Steven
Perhaps we need a HOW-TO on working with files that discusses special-case needs and solutions that use os, tempfile, etc alternatives to builtin open.
Not a bad idea, but would it not get too complicated? Starting with ways to use os.open is fine, but to do it properly we would need to go on to copying and moving, and this would involve basically rewriting the implementation of shutils into the HOW-TO. I think that this is a good idea, but not enough on its own. My suggestion, to do this properly, would be to first implement the open create mode in open. Then update (or write new) copy functions (i.e. copyfile, copy2, copytree) to use the create mode, thereby effectively changing the linux shell commands represented from "cp" to "cp -n". Finally implement a new move function which, if possible, uses link/unlink or the immutable attribute (as discussed in the tread on shutil.move), and falls back to copy/unlink using the new copytree. The resulting functions would correspond to the "no-clobber" versions of the equivalent shell commands. The only problem is that O_EXCL may not be supported on all platforms. Can anyone tell me which platforms these are? I would like to see if I can find a way to achieve the same effect on those platforms, but so far I haven't been able to find out what they are. A HOW-TO would be useful to discuss other methods, such as tempfile, which will be a lot easier to use with the no-clobber versions. David
On Thu, 18 Aug 2011 07:54:32 +0200 David Townshend <aquavitae69@gmail.com> wrote:
The only problem is that O_EXCL may not be supported on all platforms. Can anyone tell me which platforms these are?
That sounds unlikely. O_EXCL is a POSIX standard. It is also supported under Windows by the _open/_wopen compatibility functions (which we use for file I/O). Probably there are very old systems which don't support it, and perhaps new systems that don't implement it *correctly* (meaning not atomically); for the former I'd say we just don't care (who's gonna run Python 3 on a 1995 system?) and for the latter, well, if the OS designers think it's fine, let's just expose it as it is. As for NFS, there's an interesting comment from 2007 here: http://lwn.net/Articles/251971/ “My NFS tester shows that it at least appears to work with Linux, Solaris and FreeBSD: http://www.dovecot.org/list/dovecot/2007-July/024102.html. Looking at Linux 2.6 sources it doesn't look like it tries to implement a racy O_EXCL check in client side (fs/nfs/nfs3proc.c nfs3_proc_create()), so the test's results should be correct. I don't know if other OSes do that. I guess it would be nice to have a better O_EXCL tester which tries to catch race conditions.” Regards Antoine.
On Wed, Aug 17, 2011 at 1:43 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
FWIW, when you control the filename, you can include an additional subdirectory precisely for the exception when a second process attempts to create the same subdirectory. You can even play games along those lines for file access control on arbitrary filenames via a shadow hierarchy of directories. For example, Skip's lockfile package (http://packages.python.org/lockfile/lockfile.html) uses directories to provide cooperative file locks on Windows. That only helps the cooperative locking case, though - it does nothing against hostile file substitutions. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Antoine Pitrou wrote:
On Windows, directories are created atomically. On Unix, too, but symlinks are faster. You can use those to implement cooperative file locks in a fairly cross-platform way. See e.g. mx.Misc.FileLock in http://www.egenix.com/products/python/mxBase/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 17 2011)
2011-10-04: PyCon DE 2011, Leipzig, Germany 48 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Le mercredi 17 août 2011 à 11:52 +0200, M.-A. Lemburg a écrit :
I was thinking of creating both the directory and the file in a single atomic operation. But if the directory is only ever used for that file, I guess it's ok. (there's still a problem when deleting the directory and the file, which can't be atomic, and the file has to be deleted before the directory, meaning if the process crashes in between, there are "legitimate" situations where the directory exists but not the file in it...) Regards Antoine.
Antoine Pitrou wrote:
The directory is only used as locking mechanism. You normally don't need to create any files within that lock directory unless you want to store extra lock information. The lock directory can be create alongside the file you want to lock or in a separate directory (on the same file system). This mechanism can also be used to create directory/file pairs - simply lock the directory (using a separate lock directory), create the file, do something, remove the file, remove the directory, remove lock. If your process fails, you can use the information from the lock directory to implement timeouts and cleanup actions (which would then also remove the files in the directory you locked). If you additionally add a lock info file to the lock directory, you can make the checks even more sophisticated and check whether the owning process still exists, the host owning the lock is still available, etc. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 17 2011)
2011-10-04: PyCon DE 2011, Leipzig, Germany 48 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, Aug 17, 2011 at 8:19 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Ah, I thought Nick proposed to create the file in that directory. My bad.
I did (although I noted the file could be outside the directory, too). For any cooperative locking system based on persistent artifacts, crashing processes that don't release the locks properly are a definite problem (often dealt with by punting the problem to a human via an appropriate error message). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 16, 2011 at 11:45 AM, David Townshend <aquavitae69@gmail.com> wrote:
While there are plenty of responders debating how unix-y or sensible this is from the standpoint of other languages or precedents, I think this makes sense from the standpoint of new users and a fresh viewpoints. Creating a new file is a very sensible thing to want in a single operation.
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
On Wed, Aug 17, 2011 at 12:10 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Yes, this suggestion came out of the shutil.move discussion, precisely *because* it is so hard to write a platform-independent, multi-process safe operation that will reliably either create a new file or else throw an exception if the file already exists, or if the underlying filesystem doesn't provide the necessary primitives to provide the appropriate guarantees. I don't think it needs to be added to the open builtin, but a shutil function specifically for this operation certainly sounds like a reasonable request. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 17 Aug 2011 13:42:59 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Platform-independent is not hard. O_EXCL is specified by POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html Works under Linux:
Works under Windows:
(yes, I'm taking the opportunity to showcase PEP 3151) Regards Antoine.
David Townshend <aquavitae69@...> writes:
This idea was proposed in the discussion on shutil.move, but I thought it
would be worth posting separately to avoid confusing the discussion.
The idea is to add a create mode ('c') to the builtin open() function, which
will have the same effect as os.open(file, os.O_EXCL|os.O_CREAT). I have added an issue (http://bugs.python.org/issue12760) for this, including a patch. I am -1 because - Possibly not portable or at least subject to implementations of varying quality. - No precedence in other languages or fopen() for that matter. - It's not hard to use os.fdopen().
On Tue, Aug 16, 2011 at 2:26 PM, Benjamin Peterson <benjamin@python.org> wrote:
Agreed. Also I think that in most cases the right thing to do is to quietly overwrite the file. If you're implementing a UI where you want look-before-you-leap, the app should code an explicit test so it can issue a proper error message (the exception will not be fun for the user :-). -- --Guido van Rossum (python.org/~guido)
On Tue, Aug 16, 2011 at 3:43 PM, Guido van Rossum <guido@python.org> wrote:
This isn't look before you leap. It's catching a failure when the file can't be created, just as you would if you don't have permission to write a file in that directory or the file already exists and is locked so you don't have permission to overwrite it. It just adds one more reason the file can't be written. The app should already have code to translate those exceptions into human-readable error messages so I don't think that's a good objection. (If the implementation of this function in some OS needs LBYL then I think that's an unfortunate defect in those OS.) I've had enough working with programs that do things like silently eat exceptions and I consider silently overwriting a file in the same class. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On Tue, Aug 16, 2011 at 4:18 PM, Bruce Leban <bruce@leapyear.org> wrote:
So what's the use case? In general when using a command line environment overwriting the file is what you *want* to happen. Like with Unix "foo >bar". I don't even think there *is* a shell syntax for not overwriting an existing file, though you can use >> to append instead of overwrite -- this is open(filename, 'a').
Agreed.
I've had enough working with programs that do things like silently eat exceptions and I consider silently overwriting a file in the same class.
Always? How would you update an existing file if you can't overwrite files? -- --Guido van Rossum (python.org/~guido)
On Tue, Aug 16, 2011 at 4:46 PM, Guido van Rossum <guido@python.org> wrote:
We weren't just discussing command line tools. That said, I'm sure I'm not the only person on this list who has inadvertently overwritten a file using foo > bar. While we may have grown accustomed to this behavior and some people (like you) may even consider it desirable, not everyone does.
I didn't say never overwrite. What I don't like is programs overwriting files without explicitly intending to do that. Yes, there's a long legacy of overwriting files without warning or intent. I suppose I'm fighting an uphill battle (and it's not my highest priority complaint about bad code for that matter). --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com
On Tue, Aug 16, 2011 at 5:12 PM, Bruce Leban <bruce@leapyear.org> wrote:
Well if anything, Python is *lower* level than command line tools, not higher.
But are they right?
Ah, but when you write open(fn, 'w') you *do* explicitly intend to overwrite. That's what it does.
Right. -- --Guido van Rossum (python.org/~guido)
Guido van Rossum wrote:
On Tue, Aug 16, 2011 at 5:12 PM, Bruce Leban <bruce@leapyear.org> wrote:
No. I rarely intend to over-write an existing file. Usually I intend to create a new file. So I nearly always do this: if not os.path.exists(fn): open(fn, 'w') in the full knowledge that there's a race condition there, and that if I have multiple processes writing to files at the same time, as I often do, I could very well lose data. And don't think for one second I'm even close to happy about that, but at least it is *some* protection, even if not very much.
I'm not sure that a "create" mode is the right solution, but I am sure that, just like Bruce, I want a good, battle-hardened, standard, platform independent (as much as possible) solution to the above race condition bug. Perhaps it should be a module, like the tempfile module. (There's a thread about adding something like this to shutil.) -- Steven
Guido van Rossum <guido@python.org> writes:
So what's the use case?
I agree that I don't see the use case that isn't already adequately covered. You generally get what you explicitly ask for, and that's how it should be. A point of correction, though:
The ‘bash(1)’ man page describes the ‘noclobber’ option: If the redirection operator is >, and the noclobber option to the set builtin has been enabled, the redirection will fail if the file whose name results from the expansion of word exists and is a regular file. -- \ “If you go flying back through time and you see somebody else | `\ flying forward into the future, it's probably best to avoid eye | _o__) contact.” —Jack Handey | Ben Finney
The explicit test being "if not os.path.exists(pth)"? That has a race condition. LBYL is impossible here without a race condition, in fact, because the situation can change between looking and leaping. An exception, or else return code, is the only way. These can be checked for after the fact. I'd also point out that for those that don't want race conditions, Python is discouraging. The correct incantation involves two undocumented constants, plus a unique and rarely used way of opening files that involves unix file descriptors. Devin On Tue, Aug 16, 2011 at 6:43 PM, Guido van Rossum <guido@python.org> wrote:
Devin Jeanpierre <jeanpierreda@...> writes:
How often in this used? In every application I've written, writing a file usually results from the user giving a path, in which case it's intended to replace whatever is already there.
If you truly want to avoid all filesystem race conditions, you're going to be dealing with file descriptors and low-level syscalls galore. Moving one aspect to a higher level is not too helpful on the whole.
On Tue, Aug 16, 2011 at 11:49 PM, Benjamin Peterson <benjamin@python.org> wrote:
Unless it isn't. Most GUI apps ask you to confirm whether you want to open a file even where one exists. Suppose you do a LBYL approach: you check to see if no file is there, then somebody writes a file there, then you overwrite it because you used 'w' mode because no file was there. It's not a disaster, since this is kind of hard to do by accident, but it is incorrect behavior if you wanted to actually ask if overwriting was kosher. On the other hand, if you ask _after_ trying to open the file, then one of two things can happen: the user says "abort", or the user says, "overwrite it". In the former case, we start over. In the latter case, the only remaining race condition is one that doesn't matter: the file might disappear before you overwrite it!
Well, eh, not really. As far as I know this particular primitive is the probably the most important one. It's certainly the only one I've ever wanted to use Devin
It seems that the only problem raised which hasn't been already responded to is that it won't necessarily work on all platforms. Could anyone elaborate on which platforms it won't work on? From what I've read it will work on all unix and windows systems which I think covers the majority. I can't see that it would be a problem if it doesn't work on a few specialised systems as long as the documentation is clear on this (any it might turn out that it can be handled, just through os-specific code). It at least allows the feature to be available for the 90% of use cases when it will work. David On Wed, Aug 17, 2011 at 6:13 AM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
Benjamin Peterson wrote:
I frequently write scripts that, e.g. rename or move files with little or no human intervention. wget is a prime example of an application which needs to avoid overwriting files without human intervention, although I don't know how well it deals with race conditions. -- Steven
Perhaps we need a HOW-TO on working with files that discusses special-case needs and solutions that use os, tempfile, etc alternatives to builtin open.
Not a bad idea, but would it not get too complicated? Starting with ways to use os.open is fine, but to do it properly we would need to go on to copying and moving, and this would involve basically rewriting the implementation of shutils into the HOW-TO. I think that this is a good idea, but not enough on its own. My suggestion, to do this properly, would be to first implement the open create mode in open. Then update (or write new) copy functions (i.e. copyfile, copy2, copytree) to use the create mode, thereby effectively changing the linux shell commands represented from "cp" to "cp -n". Finally implement a new move function which, if possible, uses link/unlink or the immutable attribute (as discussed in the tread on shutil.move), and falls back to copy/unlink using the new copytree. The resulting functions would correspond to the "no-clobber" versions of the equivalent shell commands. The only problem is that O_EXCL may not be supported on all platforms. Can anyone tell me which platforms these are? I would like to see if I can find a way to achieve the same effect on those platforms, but so far I haven't been able to find out what they are. A HOW-TO would be useful to discuss other methods, such as tempfile, which will be a lot easier to use with the no-clobber versions. David
On Thu, 18 Aug 2011 07:54:32 +0200 David Townshend <aquavitae69@gmail.com> wrote:
The only problem is that O_EXCL may not be supported on all platforms. Can anyone tell me which platforms these are?
That sounds unlikely. O_EXCL is a POSIX standard. It is also supported under Windows by the _open/_wopen compatibility functions (which we use for file I/O). Probably there are very old systems which don't support it, and perhaps new systems that don't implement it *correctly* (meaning not atomically); for the former I'd say we just don't care (who's gonna run Python 3 on a 1995 system?) and for the latter, well, if the OS designers think it's fine, let's just expose it as it is. As for NFS, there's an interesting comment from 2007 here: http://lwn.net/Articles/251971/ “My NFS tester shows that it at least appears to work with Linux, Solaris and FreeBSD: http://www.dovecot.org/list/dovecot/2007-July/024102.html. Looking at Linux 2.6 sources it doesn't look like it tries to implement a racy O_EXCL check in client side (fs/nfs/nfs3proc.c nfs3_proc_create()), so the test's results should be correct. I don't know if other OSes do that. I guess it would be nice to have a better O_EXCL tester which tries to catch race conditions.” Regards Antoine.
On Wed, Aug 17, 2011 at 1:43 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
FWIW, when you control the filename, you can include an additional subdirectory precisely for the exception when a second process attempts to create the same subdirectory. You can even play games along those lines for file access control on arbitrary filenames via a shadow hierarchy of directories. For example, Skip's lockfile package (http://packages.python.org/lockfile/lockfile.html) uses directories to provide cooperative file locks on Windows. That only helps the cooperative locking case, though - it does nothing against hostile file substitutions. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Antoine Pitrou wrote:
On Windows, directories are created atomically. On Unix, too, but symlinks are faster. You can use those to implement cooperative file locks in a fairly cross-platform way. See e.g. mx.Misc.FileLock in http://www.egenix.com/products/python/mxBase/ -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 17 2011)
2011-10-04: PyCon DE 2011, Leipzig, Germany 48 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Le mercredi 17 août 2011 à 11:52 +0200, M.-A. Lemburg a écrit :
I was thinking of creating both the directory and the file in a single atomic operation. But if the directory is only ever used for that file, I guess it's ok. (there's still a problem when deleting the directory and the file, which can't be atomic, and the file has to be deleted before the directory, meaning if the process crashes in between, there are "legitimate" situations where the directory exists but not the file in it...) Regards Antoine.
Antoine Pitrou wrote:
The directory is only used as locking mechanism. You normally don't need to create any files within that lock directory unless you want to store extra lock information. The lock directory can be create alongside the file you want to lock or in a separate directory (on the same file system). This mechanism can also be used to create directory/file pairs - simply lock the directory (using a separate lock directory), create the file, do something, remove the file, remove the directory, remove lock. If your process fails, you can use the information from the lock directory to implement timeouts and cleanup actions (which would then also remove the files in the directory you locked). If you additionally add a lock info file to the lock directory, you can make the checks even more sophisticated and check whether the owning process still exists, the host owning the lock is still available, etc. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 17 2011)
2011-10-04: PyCon DE 2011, Leipzig, Germany 48 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, Aug 17, 2011 at 8:19 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Ah, I thought Nick proposed to create the file in that directory. My bad.
I did (although I noted the file could be outside the directory, too). For any cooperative locking system based on persistent artifacts, crashing processes that don't release the locks properly are a definite problem (often dealt with by punting the problem to a human via an appropriate error message). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 16, 2011 at 11:45 AM, David Townshend <aquavitae69@gmail.com> wrote:
While there are plenty of responders debating how unix-y or sensible this is from the standpoint of other languages or precedents, I think this makes sense from the standpoint of new users and a fresh viewpoints. Creating a new file is a very sensible thing to want in a single operation.
-- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
On Wed, Aug 17, 2011 at 12:10 PM, Calvin Spealman <ironfroggy@gmail.com> wrote:
Yes, this suggestion came out of the shutil.move discussion, precisely *because* it is so hard to write a platform-independent, multi-process safe operation that will reliably either create a new file or else throw an exception if the file already exists, or if the underlying filesystem doesn't provide the necessary primitives to provide the appropriate guarantees. I don't think it needs to be added to the open builtin, but a shutil function specifically for this operation certainly sounds like a reasonable request. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 17 Aug 2011 13:42:59 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Platform-independent is not hard. O_EXCL is specified by POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html Works under Linux:
Works under Windows:
(yes, I'm taking the opportunity to showcase PEP 3151) Regards Antoine.
participants (12)
-
Antoine Pitrou -
Ben Finney -
Benjamin Peterson -
Bruce Leban -
Calvin Spealman -
David Townshend -
Devin Jeanpierre -
Guido van Rossum -
M.-A. Lemburg -
Nick Coghlan -
Steven D'Aprano -
Terry Reedy