On Fri, Dec 25, 2020 at 9:30 AM Steven D'Aprano wrote:

Perhaps a "sync on close" keyword argument to open? At least then it is always available and easily discoverable.

+1 (though this is really just bikeshedding)

...

3. There are many ways to do this, and I think several of them could be subtly incorrect.

Can you elaborate?

I mean, the obvious way is:

try: with open(..., 'w') as f: f.write("stuff") finally: os.sync()

so maybe all we really need is a "sync file" context manager.

Does that sync the entire file system or just that one file? If there's a way to sync just one file, that would be potentially a LOT more efficient. ChrisA

On Thu, Dec 24, 2020 at 6:18 PM Cameron Simpson wrote:

On 25Dec2020 09:29, Steven D'Aprano wrote:

...

On Thu, Dec 24, 2020 at 12:15:08PM -0500, Michael A. Smith wrote:

...

With all the buffering that modern disks and filesystems do, a specific question has come up a few times with respect to whether or not data was actually written after flush. I think it would be pretty useful for the standard library to have a variant in the io module that would explicitly fsync on close.

One argument against this idea is that "disks and file systems buffer for a reason, you should trust them, explicitly calling sync after every written file is just going to slow I/O down".

Personally I don't believe this argument, I've been bitten many, many times until I learned to explicitly sync files, but its an argument you should counter.

By contrast, I support this argument. The _vast_ majority of things don't need to sync their data all the way to the hardware base substrate (eg magnetic fields on spinning rust).

And on the whole, if I do care, I issue a single sync() call at the end of a large task (typically interactively, at a prompt!) rather than forcing a heap of performance impairing stutters all the way through some process because many per-file syncs force that.

IMO, per-file syncs fall into the "policy" arena: aside from low level tools (example: fdisk, a disc partition editor), to my mind the purpose of the kernel is to accept responsibility for my data when I hand it off.

Perhaps for you that isn't enough; for me it normally is. And when it isn't, I'll take steps myself, _outside_ the programme, to ensure the sync or commit or off site backup is complete when it matters. Thus the policy is in my hands.

The tool which causes a per-file sync all on every close, or even after every write, is a performance killer. The faster our hardware, the less that may seem to matter (and, conversely, the less the risk as the ordinary kernel I/O flushing will catch up faster). But when the hardware slowness _is_ relevant, if I can't turn that off I have a needlessly unperformant task.

The example which stands out in my own mind is when I was using firefox on a laptop with a spinning rust hard drive (and being a laptop hardware, a low power physically slow piece of spinning rust). There was once a setting to turn off the synchronous-write sqlite setting (used for history and bookmarks). That was _visibly obvious_ in the user experience. And I turned it off. As a matter of policy, those data didn't need such care.

So I'm resistant to this kind of thing because IMO it leads to an attractive nuisance: over use of sync or fsync for everything. And it will usually not be exposed as policy the user can adjust/disable.

My rule of thumb:

If it can't be turned off, it's not a feature. - Karl Heuer

Are you arguing that if something is a bad idea to overuse, even if it's a good idea sometimes, then it shouldn't be allowed into Python, because someone might write a program that abuses that feature, you might end up with that program, and it would be irksome to deal with it? I'm not trying to present a straw man, but that is my genuine impression of what you said. If I got it wrong, I apologize and please help me understand what you meant.

...

Another argument is that even syncing your data doesn't mean that the data is actually written to disk, since the hardware can lie. On the other hand, I don't know what anyone can do, not even the kernel, in the face of deceitful hardware.

Aye.

But in principle, after a sync() or fsync() the kernel at least believes that. Hardware which lies, or which claims saved data without having the rresources to guarrentee it (eg a small battery to complete the writes if there's a power out) is indeed nasty.

...

...

You might be tempted to argue that this can be done very easily in Python already, so why include it in the standard io module?

I would indeed. There _should_ be a small bar which at least causes the programmer to think "do I really need this here"? I suppose a "fsync=False" default parameter is a visible bar.

[...]

...

I mean, the obvious way is:

try: with open(..., 'w') as f: f.write("stuff") finally: os.sync()

An os.fsync(f.fileno()) is lower impact - os.sync() requests a sync of all filesystems.

...

so maybe all we really need is a "sync file" context manager.

Aye. Fully agree here, and frankly think this is a "write your own" situation. Except, of course, that like all "write your own" one/few liners there will be suboptimal or buggy ones released. Such as the "overly wide sync" from your os.sync() above.

Personally I'm -1 on this. A context manager while goes f.flush() os.fsync(f.fileno()) seems plenty, and easy to roll your own.

There are very smart people on this list who have already demonstrated that there is more than one way to do it, and that it's not obvious. So, it's not easy to roll your own correctly. I love context managers when they're alone, but I dislike stacking them. It is less clear how we can ensure the fsync happens exactly between flush and close with a context manager than a keyword argument to open. That is, if open is the only context manager, everything is great. But if is up to users to stack context managers including open and some fsync, I think correct ordering will be a problem. Thank you for engaging on this topic.

Also sorry for the duplicate, Steven. On Thu, Dec 24, 2020 at 5:31 PM Steven D'Aprano wrote:

On Thu, Dec 24, 2020 at 12:15:08PM -0500, Michael A. Smith wrote:

...

With all the buffering that modern disks and filesystems do, a specific question has come up a few times with respect to whether or not data was actually written after flush. I think it would be pretty useful for the standard library to have a variant in the io module that would explicitly fsync on close.

One argument against this idea is that "disks and file systems buffer for a reason, you should trust them, explicitly calling sync after every written file is just going to slow I/O down".

Personally I don't believe this argument, I've been bitten many, many times until I learned to explicitly sync files, but its an argument you should counter.

Thanks for playing devil's advocate. I can see this argument came up in some other branches on this thread. I'll address it there. Another argument is that even syncing your data doesn't mean that the

data is actually written to disk, since the hardware can lie. On the other hand, I don't know what anyone can do, not even the kernel, in the face of deceitful hardware.

Death and taxes. The best we could do here is address it in the documentation as something else to be aware of. That would be fine with me.

3. There are many ways to do this, and I think several of them could

...

be subtly incorrect.

Can you elaborate?

I mean, the obvious way is:

try: with open(..., 'w') as f: f.write("stuff") finally: os.sync()

I think this illustrates my point. Aside from os.sync syncing more than the user might expect, I believe the sync here happens after `f` is closed. IIUC, it's safest to fsync between flush and close. Reference: https://stackoverflow.com/questions/37288453/calling-fsync2-after-close2 We'd also probably want to fdatasync() instead of fsync or os.sync if we can. so maybe all we really need is a "sync file" context manager.

I think that since open is already a context manager, putting another context manager in play is not as desirable as another keyword argument to open.

On Thu, Dec 24, 2020 at 06:06:32PM -0600, re:fi.64 wrote:

At least on Linux, this wouldn't be that ideal. These LWN articles go over the general fsync mess and the lack of viable alternatives:

https://lwn.net/Articles/351422/

But isn't that a trade-off application developers can make for themselves? Performance over reliability? We're not suggesting that fsync before close becomes mandatory or even the default. People have to opt in to the behaviour, just as they do now. -- Steve

Sorry for the duplicate, Barry. I got bit by the "I don't reply-all by default" spider. On Sat, Dec 26, 2020 at 12:30 PM Barry Scott wrote:

...

On 24 Dec 2020, at 17:15, Michael A. Smith wrote:

With all the buffering that modern disks and filesystems do, a specific question has come up a few times with respect to whether or not data was actually written after flush. I think it would be pretty useful for the standard library to have a variant in the io module that would explicitly fsync on close.

You might be tempted to argue that this can be done very easily in Python already, so why include it in the standard io module?

1. It seems to me that it would be better to do this in C, so for the folks who need to make a consistency > performance kind of choice, they don't have to sacrifice any additional performance. 2. Having it in the io library will call attention to this issue, which I think is something a lot of folks don't consider. Assuming that `close` or `flush` are sufficient for consistency has always been wrong (at its limits), but it was less likely to be a stumbling block in the past, when buffering was less aggressive and less layered and the peak size and continuous-ness of data streams was a more niche concern. 3. There are many ways to do this, and I think several of them could be subtly incorrect. In other words, maybe it can't be done very easily and correctly after all. Providing "obviously right" ways to do things is the baileywick of the standard library, isn't it?

I have used rename to make a new file appear atomically after it is closed and I have used fsync to ensure records are on disk before a file is closed.

I've not needed fsync on close yet.

I'm confused -- did you not just describe fsync-on-close? Using fsync to ensure records are on disk before a file is closed is what we're talking about. Perhaps you think I mean to fsync after close? What I really mean is 1. flush if applicable 2. (fdatasync or fsync) if indicated 3. close in that order. What is the use case that needs this? One use case is atomically writing a file by first writing to a temporary file, then renaming the temporary file to its real destination, such as the problem described by He Chen in https://issues.apache.org/jira/browse/AVRO-3013.