With all the buffering that modern disks and filesystems do, a specific question has come up a few times with respect to whether or not data was actually written after flush. I think it would be pretty useful for the standard library to have a variant in the io module that would explicitly fsync on close. You might be tempted to argue that this can be done very easily in Python already, so why include it in the standard io module? 1. It seems to me that it would be better to do this in C, so for the folks who need to make a consistency > performance kind of choice, they don't have to sacrifice any additional performance. 2. Having it in the io library will call attention to this issue, which I think is something a lot of folks don't consider. Assuming that `close` or `flush` are sufficient for consistency has always been wrong (at its limits), but it was less likely to be a stumbling block in the past, when buffering was less aggressive and less layered and the peak size and continuous-ness of data streams was a more niche concern. 3. There are many ways to do this, and I think several of them could be subtly incorrect. In other words, maybe it can't be done very easily and correctly after all. Providing "obviously right" ways to do things is the baileywick of the standard library, isn't it? Thanks for your consideration and feedback, Michael Smith
On Thu, Dec 24, 2020 at 12:15:08PM -0500, Michael A. Smith wrote:
One argument against this idea is that "disks and file systems buffer for a reason, you should trust them, explicitly calling sync after every written file is just going to slow I/O down". Personally I don't believe this argument, I've been bitten many, many times until I learned to explicitly sync files, but its an argument you should counter. Another argument is that even syncing your data doesn't mean that the data is actually written to disk, since the hardware can lie. On the other hand, I don't know what anyone can do, not even the kernel, in the face of deceitful hardware.
The actual I/O is surely going to outweigh the cost of calling sync from Python. This sounds like a trivial micro-optimization for small files, and an undetectable one for large files on slow media. If you save a dozen microseconds when syncing a two gigabyte file written to a USB-2 stick, the sync might take four or five minutes. Are you even going to notice the difference? I think you need to show benchmarks before claiming that this needs to be in C.
I don't know, I wonder whether burying it in the io library will make it disappear. Perhaps a "sync on close" keyword argument to open? At least then it is always available and easily discoverable.
3. There are many ways to do this, and I think several of them could be subtly incorrect.
Can you elaborate? I mean, the obvious way is: try: with open(..., 'w') as f: f.write("stuff") finally: os.sync() so maybe all we really need is a "sync file" context manager. -- Steve
On Fri, Dec 25, 2020 at 9:30 AM Steven D'Aprano <steve@pearwood.info> wrote:
Perhaps a "sync on close" keyword argument to open? At least then it is always available and easily discoverable.
+1 (though this is really just bikeshedding)
Does that sync the entire file system or just that one file? If there's a way to sync just one file, that would be potentially a LOT more efficient. ChrisA
On 25Dec2020 09:29, Steven D'Aprano <steve@pearwood.info> wrote:
By contrast, I support this argument. The _vast_ majority of things don't need to sync their data all the way to the hardware base substrate (eg magnetic fields on spinning rust). And on the whole, if I do care, I issue a single sync() call at the end of a large task (typically interactively, at a prompt!) rather than forcing a heap of performance impairing stutters all the way through some process because many per-file syncs force that. IMO, per-file syncs fall into the "policy" arena: aside from low level tools (example: fdisk, a disc partition editor), to my mind the purpose of the kernel is to accept responsibility for my data when I hand it off. Perhaps for you that isn't enough; for me it normally is. And when it isn't, I'll take steps myself, _outside_ the programme, to ensure the sync or commit or off site backup is complete when it matters. Thus the policy is in my hands. The tool which causes a per-file sync all on every close, or even after every write, is a performance killer. The faster our hardware, the less that may seem to matter (and, conversely, the less the risk as the ordinary kernel I/O flushing will catch up faster). But when the hardware slowness _is_ relevant, if I can't turn that off I have a needlessly unperformant task. The example which stands out in my own mind is when I was using firefox on a laptop with a spinning rust hard drive (and being a laptop hardware, a low power physically slow piece of spinning rust). There was once a setting to turn off the synchronous-write sqlite setting (used for history and bookmarks). That was _visibly obvious_ in the user experience. And I turned it off. As a matter of policy, those data didn't need such care. So I'm resistant to this kind of thing because IMO it leads to an attractive nuisance: over use of sync or fsync for everything. And it will usually not be exposed as policy the user can adjust/disable. My rule of thumb: If it can't be turned off, it's not a feature. - Karl Heuer
Aye. But in principle, after a sync() or fsync() the kernel at least believes that. Hardware which lies, or which claims saved data without having the rresources to guarrentee it (eg a small battery to complete the writes if there's a power out) is indeed nasty.
You might be tempted to argue that this can be done very easily in Python already, so why include it in the standard io module?
I would indeed. There _should_ be a small bar which at least causes the programmer to think "do I really need this here"? I suppose a "fsync=False" default parameter is a visible bar. [...]
An os.fsync(f.fileno()) is lower impact - os.sync() requests a sync of all filesystems.
so maybe all we really need is a "sync file" context manager.
Aye. Fully agree here, and frankly think this is a "write your own" situation. Except, of course, that like all "write your own" one/few liners there will be suboptimal or buggy ones released. Such as the "overly wide sync" from your os.sync() above. Personally I'm -1 on this. A context manager while goes f.flush() os.fsync(f.fileno()) seems plenty, and easy to roll your own. Cheers, Cameron Simpson <cs@cskk.id.au>
On Thu, Dec 24, 2020 at 6:18 PM Cameron Simpson <cs@cskk.id.au> wrote:
Are you arguing that if something is a bad idea to overuse, even if it's a good idea sometimes, then it shouldn't be allowed into Python, because someone might write a program that abuses that feature, you might end up with that program, and it would be irksome to deal with it? I'm not trying to present a straw man, but that is my genuine impression of what you said. If I got it wrong, I apologize and please help me understand what you meant.
There are very smart people on this list who have already demonstrated that there is more than one way to do it, and that it's not obvious. So, it's not easy to roll your own correctly. I love context managers when they're alone, but I dislike stacking them. It is less clear how we can ensure the fsync happens exactly between flush and close with a context manager than a keyword argument to open. That is, if open is the only context manager, everything is great. But if is up to users to stack context managers including open and some fsync, I think correct ordering will be a problem. Thank you for engaging on this topic.
On Sat, Dec 26, 2020 at 07:39:42PM -0500, Michael Smith wrote:
Indeed. I started playing with this idea, and even though I intellectually know that you can't sync a closed file (fsync needs to know the file descriptor ID) I keep wanting to write: with sync(): with open(...) as f: ... I dunno, maybe it's just me, and I'm sure that if I get a sensible exception on that I would soon learn to write it as: with open(...) as f: with sync(f): ... But what concerns me more is what happens in more complex cases, for example with two or more files being written: with open(..., 'w') as f: with open(..., 'w') as g: with sync(g): with sync(f): ... Aside from the deeper nesting, its not at all obvious to me whether the syncs need to be in the same or opposite order to the opens, or if it even matters. And what if people insert code around the sync? with open(...) as f: # code writing to the file here is probably safe with sync(f): ... # but code writing to the file here is not If I have to remember all these fiddly little rules for the order of context managers and what can go where, I'd rather than stick to syncing manually: with open(...) as f: write stuff sync even though I have no idea of the pros and cons of fdatasync versus fsync and will probably just forget and use os.sync. Even though I don't usually like functions with a million arguments, open is an exception. I'm +1 on adding a keyword-only "sync=False" parameter to open: - if the file is opened in read-only mode, has no effect; - if the file is opened for writing, guarantees that fdatasync or fsync will be called after flush and before closing. (I'm not entirely sure fdatasync is better than fsync, it's only available on non-OS-X Unixes and I don't know why I would want to update the file data but not the metadata.) -- Steve
Also sorry for the duplicate, Steven. On Thu, Dec 24, 2020 at 5:31 PM Steven D'Aprano <steve@pearwood.info> wrote:
Thanks for playing devil's advocate. I can see this argument came up in some other branches on this thread. I'll address it there. Another argument is that even syncing your data doesn't mean that the
Death and taxes. The best we could do here is address it in the documentation as something else to be aware of. That would be fine with me.
I think this illustrates my point. Aside from os.sync syncing more than the user might expect, I believe the sync here happens after `f` is closed. IIUC, it's safest to fsync between flush and close. Reference: https://stackoverflow.com/questions/37288453/calling-fsync2-after-close2 We'd also probably want to fdatasync() instead of fsync or os.sync if we can. so maybe all we really need is a "sync file" context manager.
I think that since open is already a context manager, putting another context manager in play is not as desirable as another keyword argument to open.
[...] most file systems take the brute force approach, with the result
So, now we have the following situation: fsync() is required to guarantee
The problem is that filesystem developers are not willing to make much in
At least on Linux, this wouldn't be that ideal. These LWN articles go over the general fsync mess and the lack of viable alternatives: https://lwn.net/Articles/351422/ that fsync() commonly takes time proportional to all outstanding writes to the file system. that file data is on stable storage, but it may perform arbitrarily poorly, depending on what other activity is going on in the file system. https://lwn.net/Articles/788938/ the way of guarantees unless applications call fsync() [...] it is a "terrible answer" because it has impacts all over the system. On Thu, Dec 24, 2020, 11:17 AM Michael A. Smith <michael@smith-li.com> wrote:
On Thu, Dec 24, 2020 at 06:06:32PM -0600, re:fi.64 wrote:
But isn't that a trade-off application developers can make for themselves? Performance over reliability? We're not suggesting that fsync before close becomes mandatory or even the default. People have to opt in to the behaviour, just as they do now. -- Steve
I have used rename to make a new file appear atomically after it is closed and I have used fsync to ensure records are on disk before a file is closed. I've not needed fsync on close yet. What is the use case that needs this? Without a compelling use case I'm -1 on this. Barry
Sorry for the duplicate, Barry. I got bit by the "I don't reply-all by default" spider. On Sat, Dec 26, 2020 at 12:30 PM Barry Scott <barry@barrys-emacs.org> wrote:
I'm confused -- did you not just describe fsync-on-close? Using fsync to ensure records are on disk before a file is closed is what we're talking about. Perhaps you think I mean to fsync after close? What I really mean is 1. flush if applicable 2. (fdatasync or fsync) if indicated 3. close in that order. What is the use case that needs this? One use case is atomically writing a file by first writing to a temporary file, then renaming the temporary file to its real destination, such as the problem described by He Chen in https://issues.apache.org/jira/browse/AVRO-3013.
On Thu, Dec 24, 2020 at 12:15:08PM -0500, Michael A. Smith wrote:
One argument against this idea is that "disks and file systems buffer for a reason, you should trust them, explicitly calling sync after every written file is just going to slow I/O down". Personally I don't believe this argument, I've been bitten many, many times until I learned to explicitly sync files, but its an argument you should counter. Another argument is that even syncing your data doesn't mean that the data is actually written to disk, since the hardware can lie. On the other hand, I don't know what anyone can do, not even the kernel, in the face of deceitful hardware.
The actual I/O is surely going to outweigh the cost of calling sync from Python. This sounds like a trivial micro-optimization for small files, and an undetectable one for large files on slow media. If you save a dozen microseconds when syncing a two gigabyte file written to a USB-2 stick, the sync might take four or five minutes. Are you even going to notice the difference? I think you need to show benchmarks before claiming that this needs to be in C.
I don't know, I wonder whether burying it in the io library will make it disappear. Perhaps a "sync on close" keyword argument to open? At least then it is always available and easily discoverable.
3. There are many ways to do this, and I think several of them could be subtly incorrect.
Can you elaborate? I mean, the obvious way is: try: with open(..., 'w') as f: f.write("stuff") finally: os.sync() so maybe all we really need is a "sync file" context manager. -- Steve
On Fri, Dec 25, 2020 at 9:30 AM Steven D'Aprano <steve@pearwood.info> wrote:
Perhaps a "sync on close" keyword argument to open? At least then it is always available and easily discoverable.
+1 (though this is really just bikeshedding)
Does that sync the entire file system or just that one file? If there's a way to sync just one file, that would be potentially a LOT more efficient. ChrisA
On 25Dec2020 09:29, Steven D'Aprano <steve@pearwood.info> wrote:
By contrast, I support this argument. The _vast_ majority of things don't need to sync their data all the way to the hardware base substrate (eg magnetic fields on spinning rust). And on the whole, if I do care, I issue a single sync() call at the end of a large task (typically interactively, at a prompt!) rather than forcing a heap of performance impairing stutters all the way through some process because many per-file syncs force that. IMO, per-file syncs fall into the "policy" arena: aside from low level tools (example: fdisk, a disc partition editor), to my mind the purpose of the kernel is to accept responsibility for my data when I hand it off. Perhaps for you that isn't enough; for me it normally is. And when it isn't, I'll take steps myself, _outside_ the programme, to ensure the sync or commit or off site backup is complete when it matters. Thus the policy is in my hands. The tool which causes a per-file sync all on every close, or even after every write, is a performance killer. The faster our hardware, the less that may seem to matter (and, conversely, the less the risk as the ordinary kernel I/O flushing will catch up faster). But when the hardware slowness _is_ relevant, if I can't turn that off I have a needlessly unperformant task. The example which stands out in my own mind is when I was using firefox on a laptop with a spinning rust hard drive (and being a laptop hardware, a low power physically slow piece of spinning rust). There was once a setting to turn off the synchronous-write sqlite setting (used for history and bookmarks). That was _visibly obvious_ in the user experience. And I turned it off. As a matter of policy, those data didn't need such care. So I'm resistant to this kind of thing because IMO it leads to an attractive nuisance: over use of sync or fsync for everything. And it will usually not be exposed as policy the user can adjust/disable. My rule of thumb: If it can't be turned off, it's not a feature. - Karl Heuer
Aye. But in principle, after a sync() or fsync() the kernel at least believes that. Hardware which lies, or which claims saved data without having the rresources to guarrentee it (eg a small battery to complete the writes if there's a power out) is indeed nasty.
You might be tempted to argue that this can be done very easily in Python already, so why include it in the standard io module?
I would indeed. There _should_ be a small bar which at least causes the programmer to think "do I really need this here"? I suppose a "fsync=False" default parameter is a visible bar. [...]
An os.fsync(f.fileno()) is lower impact - os.sync() requests a sync of all filesystems.
so maybe all we really need is a "sync file" context manager.
Aye. Fully agree here, and frankly think this is a "write your own" situation. Except, of course, that like all "write your own" one/few liners there will be suboptimal or buggy ones released. Such as the "overly wide sync" from your os.sync() above. Personally I'm -1 on this. A context manager while goes f.flush() os.fsync(f.fileno()) seems plenty, and easy to roll your own. Cheers, Cameron Simpson <cs@cskk.id.au>
On Thu, Dec 24, 2020 at 6:18 PM Cameron Simpson <cs@cskk.id.au> wrote:
Are you arguing that if something is a bad idea to overuse, even if it's a good idea sometimes, then it shouldn't be allowed into Python, because someone might write a program that abuses that feature, you might end up with that program, and it would be irksome to deal with it? I'm not trying to present a straw man, but that is my genuine impression of what you said. If I got it wrong, I apologize and please help me understand what you meant.
There are very smart people on this list who have already demonstrated that there is more than one way to do it, and that it's not obvious. So, it's not easy to roll your own correctly. I love context managers when they're alone, but I dislike stacking them. It is less clear how we can ensure the fsync happens exactly between flush and close with a context manager than a keyword argument to open. That is, if open is the only context manager, everything is great. But if is up to users to stack context managers including open and some fsync, I think correct ordering will be a problem. Thank you for engaging on this topic.
On Sat, Dec 26, 2020 at 07:39:42PM -0500, Michael Smith wrote:
Indeed. I started playing with this idea, and even though I intellectually know that you can't sync a closed file (fsync needs to know the file descriptor ID) I keep wanting to write: with sync(): with open(...) as f: ... I dunno, maybe it's just me, and I'm sure that if I get a sensible exception on that I would soon learn to write it as: with open(...) as f: with sync(f): ... But what concerns me more is what happens in more complex cases, for example with two or more files being written: with open(..., 'w') as f: with open(..., 'w') as g: with sync(g): with sync(f): ... Aside from the deeper nesting, its not at all obvious to me whether the syncs need to be in the same or opposite order to the opens, or if it even matters. And what if people insert code around the sync? with open(...) as f: # code writing to the file here is probably safe with sync(f): ... # but code writing to the file here is not If I have to remember all these fiddly little rules for the order of context managers and what can go where, I'd rather than stick to syncing manually: with open(...) as f: write stuff sync even though I have no idea of the pros and cons of fdatasync versus fsync and will probably just forget and use os.sync. Even though I don't usually like functions with a million arguments, open is an exception. I'm +1 on adding a keyword-only "sync=False" parameter to open: - if the file is opened in read-only mode, has no effect; - if the file is opened for writing, guarantees that fdatasync or fsync will be called after flush and before closing. (I'm not entirely sure fdatasync is better than fsync, it's only available on non-OS-X Unixes and I don't know why I would want to update the file data but not the metadata.) -- Steve
Also sorry for the duplicate, Steven. On Thu, Dec 24, 2020 at 5:31 PM Steven D'Aprano <steve@pearwood.info> wrote:
Thanks for playing devil's advocate. I can see this argument came up in some other branches on this thread. I'll address it there. Another argument is that even syncing your data doesn't mean that the
Death and taxes. The best we could do here is address it in the documentation as something else to be aware of. That would be fine with me.
I think this illustrates my point. Aside from os.sync syncing more than the user might expect, I believe the sync here happens after `f` is closed. IIUC, it's safest to fsync between flush and close. Reference: https://stackoverflow.com/questions/37288453/calling-fsync2-after-close2 We'd also probably want to fdatasync() instead of fsync or os.sync if we can. so maybe all we really need is a "sync file" context manager.
I think that since open is already a context manager, putting another context manager in play is not as desirable as another keyword argument to open.
[...] most file systems take the brute force approach, with the result
So, now we have the following situation: fsync() is required to guarantee
The problem is that filesystem developers are not willing to make much in
At least on Linux, this wouldn't be that ideal. These LWN articles go over the general fsync mess and the lack of viable alternatives: https://lwn.net/Articles/351422/ that fsync() commonly takes time proportional to all outstanding writes to the file system. that file data is on stable storage, but it may perform arbitrarily poorly, depending on what other activity is going on in the file system. https://lwn.net/Articles/788938/ the way of guarantees unless applications call fsync() [...] it is a "terrible answer" because it has impacts all over the system. On Thu, Dec 24, 2020, 11:17 AM Michael A. Smith <michael@smith-li.com> wrote:
On Thu, Dec 24, 2020 at 06:06:32PM -0600, re:fi.64 wrote:
But isn't that a trade-off application developers can make for themselves? Performance over reliability? We're not suggesting that fsync before close becomes mandatory or even the default. People have to opt in to the behaviour, just as they do now. -- Steve
I have used rename to make a new file appear atomically after it is closed and I have used fsync to ensure records are on disk before a file is closed. I've not needed fsync on close yet. What is the use case that needs this? Without a compelling use case I'm -1 on this. Barry
Sorry for the duplicate, Barry. I got bit by the "I don't reply-all by default" spider. On Sat, Dec 26, 2020 at 12:30 PM Barry Scott <barry@barrys-emacs.org> wrote:
I'm confused -- did you not just describe fsync-on-close? Using fsync to ensure records are on disk before a file is closed is what we're talking about. Perhaps you think I mean to fsync after close? What I really mean is 1. flush if applicable 2. (fdatasync or fsync) if indicated 3. close in that order. What is the use case that needs this? One use case is atomically writing a file by first writing to a temporary file, then renaming the temporary file to its real destination, such as the problem described by He Chen in https://issues.apache.org/jira/browse/AVRO-3013.
participants (7)
-
Barry Scott -
Cameron Simpson -
Chris Angelico -
Michael A. Smith -
Michael Smith
-
re:fi.64 -
Steven D'Aprano