best place for an atomic file API
Hi, Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. For example, it could be used by importlib to avoid races when writting bytecode files (issues #13392, #13003, #13146), or more generally by any application that wants to make sure to end up with a consistent file even in face of crash (e.g. it seems that mercurial implemented their own version). Basically the usage would be, e.g.: with AtomicFile('foo') as f: pickle.dump(obj, f) or with AtomicFile('foo') as f: chunk = heavyCrunch() f.write(chunk) chunk = CrunchSomeMore() f.write(chunk) What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile Cheers, cf
Charles-François Natali wrote:
Hi,
Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. [...] What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile
shutil perhaps? As a user, that's the third place I look for file utilities, after builtin functions and os module. -- Steven
On Thu, Feb 16, 2012 at 9:29 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Charles-François Natali wrote:
What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile
shutil perhaps?
As a user, that's the third place I look for file utilities, after builtin functions and os module.
+1 for shutil from me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Charles-François Natali <neologix@free.fr> writes:
Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk.
These make it quite OS-specific. […]
What would be the best place for a such a class? _pyio, tempfile, or a new atomicfile
I would expect to find it within ‘os’ or submodules of ‘os’. -- \ “We should be less concerned about adding years to life, and | `\ more about adding life to years.” —Arthur C. Clarke, 2001 | _o__) | Ben Finney
On Wed, Feb 15, 2012 at 19:19, Ben Finney <ben+python@benfinney.id.au> wrote:
Charles-François Natali <neologix@free.fr> writes:
Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk.
These make it quite OS-specific.
That'll happen when solving problems on different OSes. Do you propose a more platform agnostic solution?
Brian Curtin <brian@python.org> writes:
On Wed, Feb 15, 2012 at 19:19, Ben Finney <ben+python@benfinney.id.au> wrote:
Charles-François Natali <neologix@free.fr> writes:
[…] using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk.
These make it quite OS-specific.
That'll happen when solving problems on different OSes. Do you propose a more platform agnostic solution?
No, I have no objection to that implementation. I'm pointing that out only because the nature of the functionality implies I'd expect to find it within the ‘os’ module hierarchy. -- \ “The man who is denied the opportunity of taking decisions of | `\ importance begins to regard as important the decisions he is | _o__) allowed to take.” —C. Northcote Parkinson | Ben Finney
On Thu, Feb 16, 2012 at 11:49 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
No, I have no objection to that implementation. I'm pointing that out only because the nature of the functionality implies I'd expect to find it within the ‘os’ module hierarchy.
The (very) rough rule of thumb is that the os module handles abstracting away cross-platform differences in implementation details, while the higher level shutil algorithms can be largely platform independent by using the shared abstractions in the os module layer. In this case, os.replace() is the cross platform abstraction, while the atomic file context manager is just a particular use case for that new feature. (MvL complained in the tracker issue about a lack of concrete use cases, but I think fixing race conditions when overwriting bytecode files in importlib and the existing distutils/packaging use cases cover that) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
(MvL complained in the tracker issue about a lack of concrete use cases, but I think fixing race conditions when overwriting bytecode files in importlib and the existing distutils/packaging use cases cover that)
I certainly agree that there are applications of "atomic replace", and that the os module should expose the relevant platform APIs where available. I'm not so sure that "atomic writes" is a useful concept. I haven't seen a proposed implementation, yet, but I'm doubtful that truly ACID writes are possible unless the operating system supports transactions (which only Windows 7 does). Even if you are ignoring Isolation, Atomic already is a challenge: if you first write to a tempfile, then rename it, you may end up with a state tempfile (e.g. if the process is killed), and no rollback operation. So "atomic write" to me promises something that it likely can't deliver. OTOH, I still think that the promise isn't actually asked for in practice (not even when overwriting bytecode files) Regards, Martin
Most users don't need a truly ACID write, but implement their own best-effort function. Instead of having a different implement in each project, Python can provide something better, especially when the OS provides low level function to implement such feature. Victor 2012/2/16 "Martin v. Löwis" <martin@v.loewis.de>:
I'm not so sure that "atomic writes" is a useful concept. I haven't seen a proposed implementation, yet, but I'm doubtful that truly ACID writes are possible unless the operating system supports transactions (which only Windows 7 does). Even if you are ignoring Isolation, Atomic already is a challenge: if you first write to a tempfile, then rename it, you may end up with a state tempfile (e.g. if the process is killed), and no rollback operation.
So "atomic write" to me promises something that it likely can't deliver. OTOH, I still think that the promise isn't actually asked for in practice (not even when overwriting bytecode files)
Am 16.02.2012 10:54, schrieb Victor Stinner:
Most users don't need a truly ACID write, but implement their own best-effort function. Instead of having a different implement in each project, Python can provide something better, especially when the OS provides low level function to implement such feature.
It's then critical how this is named, IMO (and exactly what semantics it comprises). Calling it "atomic" when it is not is a mistake. Also notice that one user commented that that he already implemented something like this, and left out the issue of *permissions*. I found that interesting, since preserving permissions might indeed a requirement in a lot of "in-place update" use cases, but hasn't been considered in this discussion yet. So rather than providing a mechanism for atomic writes, I think providing a mechanism to update a file is what people might need. One way of providing this might be a "u" mode for open, which updates an existing file on close (unlike "a", which appends, and unlike "w", which truncates first). Regards, Martin
15.02.12 23:16, Charles-François Natali написав(ла):
Issue #8604 aims at adding an atomic file API to make it easier to create/update files atomically, using rename() on POSIX systems and MoveFileEx() on Windows (which are now available through os.replace()). It would also use fsync() on POSIX to make sure data is committed to disk. For example, it could be used by importlib to avoid races when writting bytecode files (issues #13392, #13003, #13146), or more generally by any application that wants to make sure to end up with a consistent file even in face of crash (e.g. it seems that mercurial implemented their own version).
What if target file is symlink?
participants (9)
-
"Martin v. Löwis" -
Ben Finney -
Brian Curtin -
Charles-François Natali -
Nick Coghlan -
Serhiy Storchaka -
Steven D'Aprano -
Victor Stinner -
Vinay Sajip