wait time [was: Ext4 data loss]
It is starting to look as though flush (and close?) should take an optional wait parameter, to indicate how much re-assurance you're willing to wait for. It also looks like we can't know enough to predict all sensible symbolic constants -- so instead use a floating point numeric value. f.flush(wait=0) ==> current behavior f.flush(wait=1) ==> Do everything you can. On a Mac, this would apparently mean (everything up to and including) fcntl(fd, F_FULLSYNC) f.flush(wait=0.5) ==> somewhere in between, depending on the operating system and file system and disk drive and other stuff the devoloper won't know in advance. The exact interpretation of intermediate values might depend on the installation or even change over time; the only invariant would be that higher values are at least as safe, and lower values are at least as fast. -jJ
On Thu, 12 Mar 2009 at 20:56, "Martin v. L�wis" wrote:
It is starting to look as though flush (and close?) should take an optional wait parameter, to indicate how much re-assurance you're willing to wait for.
Unfortunately, such a thing would be unimplementable on most of today's operating systems.
I read Jim's suggestion as a way to indicate the strength of the desire of the application programmer for certainty, not as a time value. In other words, 0.0 would map to 'just flush it', 0.5 might map to 'fsync', and 1.0 map to OS-X's "tell the disk to flush its buffers' call. Assuming I'm right, I don't like the proposal. It feels too squishy: the semantics are not well defined. By the way, I would not like to see python programmers encouraged to make the same mistake that sqlite3 made. The decision about how aggressive to be on flushing data to disk should be in the hands of the _user_, not the application. Of course, the application needs some way to enable the user to make that decision, which is what I presume we are talking about supporting here. -- R. David Murray http://www.bitdance.com
R. David Murray <rdmurray <at> bitdance.com> writes:
By the way, I would not like to see python programmers encouraged to make the same mistake that sqlite3 made. The decision about how aggressive to be on flushing data to disk should be in the hands of the _user_, not the application.
I disagree. The user usually does not know which kind of flushing is needed in order for his data to be safe. Actually, he probably doesn't even know what flushing means, and that files are ever "closed". However, I also think that any parameter to flush() or close() is a bad idea, since it can't be used when flushing and closing is implicit. For example when the file is used in a "with" statement.
On Thu, 12 Mar 2009 at 20:25, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
By the way, I would not like to see python programmers encouraged to make the same mistake that sqlite3 made. The decision about how aggressive to be on flushing data to disk should be in the hands of the _user_, not the application.
I disagree. The user usually does not know which kind of flushing is needed in order for his data to be safe. Actually, he probably doesn't even know what flushing means, and that files are ever "closed".
Let me try some examples. Suppose I'm running my applications on a laptop and I don't want the disk to be spinning continually while I work. I'm willing to take the risk of data loss in order to extend my battery life. And then there's the high performance server situation, where all the hardware is at least double redundancy, and we want the fastest disk performance possible, with data reliability being taken care of by the redundancy in the system. (Is this actually possible with today's hardware and software? I don't know, but it _should_ be.) In between there is the medium to low performance, non-redundant server, where we are willing to trade performance for data integrity. In all three of these situations I might be running the exact same application software. So, the user needs to be in control. Of course, for users who don't understand the tradeoffs, there should be a sane default. Oh, and the user doesn't need to understand flushing, they just need to be in control of the performance versus data-integrity- in-the-face-of-crashes tradeoff. -- R. David Murray http://www.bitdance.com
Let me try some examples.
Suppose I'm running my applications on a laptop and I don't want the disk to be spinning continually while I work. I'm willing to take the risk of data loss in order to extend my battery life.
So when you select "Save" in your application, would you like the data to be saved, or would you accept that they get lost? If the latter, what kind of interaction would you perform with your application to indicate that you *do* want the data to appear on disk? Regards, Martin
On Thu, Mar 12, 2009 at 4:09 PM, "Martin v. Löwis" <martin@v.loewis.de>wrote:
So when you select "Save" in your application, would you like the data to be saved, or would you accept that they get lost? If the latter, what kind of interaction would you perform with your application to indicate that you *do* want the data to appear on disk?
I accept that if the computer crashes at just the wrong moment as I click Save, my changes will not actually be Saved. No amount of diligence in the implementation of close() can prevent that since the computer can crash before the program calls close(). I oppose applications that lose or corrupt both my new save and my *previous* save if the computer crashes at the wrong moment. That would cause me to lose not only my most recent changes (an inconvenience), but also all the work I have ever done on the file (a major headache for anyone who doesn't make regular backups). However, defaulting to calling fsync() when closing a file will: 1) Cripple performance for the many applications that don't need it (e.g., temporary files) 2) Fail to prevent data loss for applications that use the truncate-and-rewrite paradigm for saving Consider the following example: with open('mysavefile', 'w') as f: f.write(data) f.flush() os.fsync(f.fileno()) f.close() If the system crashes after the call to open(), but before the call to fsync(), then both the old and the new mysavefile may be gone. Since needing to safely replace a file with new data is a moderately common task, perhaps it would be useful to have a convenience class that looks like a file, but takes care of the ugly details behind-the-scenes? Something vaguely like this flawed and untested class: class open_for_safe_replacement(file): # needs a better name def __init__(self, path, flags): if 'w' not in flags: raise RuntimeError, 'Writing without writing?' self.path = path self.tmp_name = some_function_that_generates_a_safe_temporary_filename() # good luck file.__init__(self.tmp_name, flags) def close(self): self.flush() os.fsync(self.fileno()) self.close() os.rename(self.tmp_name, self.path) # won't work on Windows :-( then we could simply: with appropriate_module.open_for_safe_replacement('mysavefile', 'w'): f.write(data) -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
On 12Mar2009 22:09, Martin v. L?wis <martin@v.loewis.de> wrote: | > Let me try some examples. | > Suppose I'm running my applications on a laptop and I don't want the | > disk to be spinning continually while I work. I'm willing to take the | > risk of data loss in order to extend my battery life. | | So when you select "Save" in your application, would you like the data | to be saved, or would you accept that they get lost? Often, I will accept that they get lost. Why? Because that will only happen with and OS/hardware failure, and I expect those to be close to never. | If the latter, | what kind of interaction would you perform with your application to | indicate that you *do* want the data to appear on disk? I don't. I type "sync" to a convenient shell prompt. On a UNIX OS, that will not return until all outstanding data at the time of issuing the command have been commited to disc. As you can see, that places the timing in the hands of the user. Where it belongs, not impacting the performance of the system except at my own command. I speak as one who keeps his bogofilter spam datbase on a RAM disc because bogofilter, too, is subject to atrocious sync overuse, since it uses a database library that overuses sync. Testing shows at least one and possibly more _orders_of_magnitude_ improvement in behaviour. Every so often I copy the bogofilter db back to real disc. The wholse point of a good OS on decent hardware is that one can commit data to the _OS_, and trust that it will reach the disc in due course. Fsync shows an app that doesn't trust the OS. I hope you don't believe that handing the data to the disc drive guarentees it has made it to the magnetic medium. It should do, but the drive will probably acknowledge the data before the medium has completed updating. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Isaac Asimov once remarked that friends had chided him for not patenting the electronic pocket calculator, since he wrote of similar devices back in the 1940's. His reply, "Have you ever noticed I only described what it looked like on the *outside*?" - ijl@mediaone.net
Cameron Simpson wrote:
On 12Mar2009 22:09, Martin v. L?wis <martin@v.loewis.de> wrote: | > Let me try some examples. | > Suppose I'm running my applications on a laptop and I don't want the | > disk to be spinning continually while I work. I'm willing to take the | > risk of data loss in order to extend my battery life. | | So when you select "Save" in your application, would you like the data | to be saved, or would you accept that they get lost?
Often, I will accept that they get lost. Why? Because that will only happen with and OS/hardware failure, and I expect those to be close to never.
I think you are an atypical user, then. People can accept that data is lost if the machine crashes at the moment of saving. They get certainly puzzled if the data is lost if the machine crashes 30 seconds after they have saved, and not even a backup copy is available anymore. Regards, Martin
On Thu, 12 Mar 2009 at 22:57, "Martin v. L�wis" wrote:
Cameron Simpson wrote:
On 12Mar2009 22:09, Martin v. L?wis <martin@v.loewis.de> wrote: | > Let me try some examples. | > Suppose I'm running my applications on a laptop and I don't want the | > disk to be spinning continually while I work. I'm willing to take the | > risk of data loss in order to extend my battery life. | | So when you select "Save" in your application, would you like the data | to be saved, or would you accept that they get lost?
Often, I will accept that they get lost. Why? Because that will only happen with and OS/hardware failure, and I expect those to be close to never.
I think you are an atypical user, then. People can accept that data is lost if the machine crashes at the moment of saving. They get certainly puzzled if the data is lost if the machine crashes 30 seconds after they have saved, and not even a backup copy is available anymore.
The typical user is probably not all that surprised when Windows loses their data. They probably figure Windows took more than 30 seconds to complete the save :) :) Seriously, though, the point is that IMO an application should not be calling fsync unless it provides a way for that behavior to be controlled by the user. -- R. David Murray http://www.bitdance.com
R. David Murray <rdmurray <at> bitdance.com> writes:
Seriously, though, the point is that IMO an application should not be calling fsync unless it provides a way for that behavior to be controlled by the user.
But whether an application does it or not is none of Python's business, is it? What is the disagreement exactly?
On 13Mar2009 00:35, Antoine Pitrou <solipsis@pitrou.net> wrote: | R. David Murray <rdmurray <at> bitdance.com> writes: | > Seriously, though, the point is that IMO an application should not be | > calling fsync unless it provides a way for that behavior to be controlled | > by the user. | | But whether an application does it or not is none of Python's business, is it? | What is the disagreement exactly? When the app is written in python, it bears on python's business. The dispute seems to me to be largely (a) should python libraries call fsync() and the like on their own, and when (b) whether there should be class methods to control this. For myself, the answer for (a) is broadly no and for (b) preferably yes, in which case my answer to (a) becomes "default to no fsyncness unless asked". Then the behaviour of the app becomes something to criticise or not and python can go in its was with a clear conscience. Cheers, -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ DRM: the functionality of refusing to function. - Richard Stallman
On Fri, 13 Mar 2009 at 00:35, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
Seriously, though, the point is that IMO an application should not be calling fsync unless it provides a way for that behavior to be controlled by the user.
But whether an application does it or not is none of Python's business, is it? What is the disagreement exactly?
I'd like to see whatever feature gets added support the application writer in making this user controllable, or at the very least document that this to do so is best practice if you use the sync feature. -- R. David Murray http://www.bitdance.com
On Fri, 13 Mar 2009 01:02:26 pm R. David Murray wrote:
On Fri, 13 Mar 2009 at 00:35, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
Seriously, though, the point is that IMO an application should not be calling fsync unless it provides a way for that behavior to be controlled by the user.
But whether an application does it or not is none of Python's business, is it? What is the disagreement exactly?
I'd like to see whatever feature gets added support the application writer in making this user controllable, or at the very least document that this to do so is best practice if you use the sync feature.
It's not best practice. It may be best practice for a certain class of users and applications, e.g. those who value the ability to control low-level behaviour of the app, but it is poor practice for other classes of users and applications. Do you really think that having Minefield make the file syncing behaviour of the high scores file user-configurable is best practice? People care about their high scores, but they don't care that much. It may even lead to more data loss than leaving it out: * If the application chooses a specific strategy, this strategy might (for the sake of the argument) lead to data loss once in ten million writes on average. * If the application makes this a configuration option, the increased complexity of writing the code, and the increased number of paths that need to be tested, may lead to bugs which cause data loss. This may be more risky than the original strategy above (whatever that happens to be.) Complexity is not cost-free, and insisting that the more complex, expensive solution is always "best practice" is wrong. -- Steven D'Aprano
Steven D'Aprano wrote:
On Fri, 13 Mar 2009 01:02:26 pm R. David Murray wrote:
On Fri, 13 Mar 2009 at 00:35, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
Seriously, though, the point is that IMO an application should not be calling fsync unless it provides a way for that behavior to be controlled by the user. But whether an application does it or not is none of Python's business, is it? What is the disagreement exactly? I'd like to see whatever feature gets added support the application writer in making this user controllable, or at the very least document that this to do so is best practice if you use the sync feature.
It's not best practice. It may be best practice for a certain class of users and applications, e.g. those who value the ability to control low-level behaviour of the app, but it is poor practice for other classes of users and applications. Do you really think that having Minefield make the file syncing behaviour of the high scores file user-configurable is best practice? People care about their high scores, but they don't care that much.
It may even lead to more data loss than leaving it out:
* If the application chooses a specific strategy, this strategy might (for the sake of the argument) lead to data loss once in ten million writes on average.
* If the application makes this a configuration option, the increased complexity of writing the code, and the increased number of paths that need to be tested, may lead to bugs which cause data loss. This may be more risky than the original strategy above (whatever that happens to be.)
Complexity is not cost-free, and insisting that the more complex, expensive solution is always "best practice" is wrong.
If the pops and moms uses a financial program and lost their only copy of 10 years worth of financial data, they'll simply be confused and that's it. Meanwhile if a network administrator needs to squeeze the last bit of performance out of his backup script, he definitely would threaten the dev-team of the programming language to make manual sync file writing the default, since it makes it difficult for him to fine-tune the syncing process.
On Fri, 13 Mar 2009 at 14:27, Steven D'Aprano wrote:
On Fri, 13 Mar 2009 01:02:26 pm R. David Murray wrote:
On Fri, 13 Mar 2009 at 00:35, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
Seriously, though, the point is that IMO an application should not be calling fsync unless it provides a way for that behavior to be controlled by the user.
But whether an application does it or not is none of Python's business, is it? What is the disagreement exactly?
I'd like to see whatever feature gets added support the application writer in making this user controllable, or at the very least document that this to do so is best practice if you use the sync feature.
It's not best practice. It may be best practice for a certain class of users and applications, e.g. those who value the ability to control low-level behaviour of the app, but it is poor practice for other classes of users and applications. Do you really think that having Minefield make the file syncing behaviour of the high scores file user-configurable is best practice? People care about their high scores, but they don't care that much.
Why would Minefield bother to use sync/fsync? You will note that what I suggested was that applications that _use the sync feature_ make it user controllable. And yes, it is Best Practice, because we have glaring counterexamples (sqlite3, the one mentioned by another poster) that require people to jump through hoops to compensate for the fact that it isn't user controllable. I thought that a significant majority of applications wouldn't have to care. If most applications do really have to care (ie: there's no way for them to recover gracefully from a crash that trashes the new data they were trying to save unless they call fsync), then my argument is a lot weaker. And from the post that talked about the problems with rename, I gather I was misunderstanding the extent of the problem. Perhaps I can restrict my request: that it be noted that if you use fsync _aggressively_ (ie: not just on a final save that happens infrequently) that you make it user controllable in some fashion. Note that that user control could be as simple as being able to set the autosave interval on an autosaving editor if the autosave needs to do an fsync for safety. Thinking about it further, I think what I'm really looking for is a warning to applications developers that using fsync can have significant performance penalties (and therefore if they use it a lot they should make it configurable), and can make laptop users hate you if it gets used any time other than at a user requested save point :) -- R. David Murray http://www.bitdance.com
R. David Murray <rdmurray <at> bitdance.com> writes:
You will note that what I suggested was that applications that _use the sync feature_ make it user controllable.
I'm sorry, but if it has nothing to do with Python itself, perhaps we could stop this subthread (or move it to another ML)? There are enough messages already. Thanks Antoine.
On Fri, 13 Mar 2009 at 12:27, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
You will note that what I suggested was that applications that _use the sync feature_ make it user controllable.
I'm sorry, but if it has nothing to do with Python itself, perhaps we could stop this subthread (or move it to another ML)? There are enough messages already.
It has to do with python in that it has to do with how this proposed new feature gets documented. But I agree, we've gone on too long about it already. I'll make a note to myself to bring this up again (only if warranted) when the feature actually lands and I can see how it is documented. -- R. David Murray http://www.bitdance.com
On Fri, Mar 13, 2009, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
You will note that what I suggested was that applications that _use the sync feature_ make it user controllable.
I'm sorry, but if it has nothing to do with Python itself, perhaps we could stop this subthread (or move it to another ML)? There are enough messages already.
<blink> Yes, this *is* about Python: how should Python support David's use-case? This discussion started because Python currently doesn't have good mechansisms for fine-grained synching, and people are discussing how Python should support various use-cases. Please don't be too aggressive about labeling discussion off-topic. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "All problems in computer science can be solved by another level of indirection." --Butler Lampson
Aahz <aahz <at> pythoncraft.com> writes:
On Fri, Mar 13, 2009, Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
You will note that what I suggested was that applications that _use the sync feature_ make it user controllable.
I'm sorry, but if it has nothing to do with Python itself, perhaps we could stop this subthread (or move it to another ML)? There are enough messages already.
<blink> Yes, this *is* about Python: how should Python support David's use-case?
Python already has plenty of mechanisms for enabling configuration choices, so I don't see your point: the use case is /already/ supported assuming there is a sync() method at all. The discussion is (or should be) about whether/how Python should expose a sync() facility in its file input/output stack (or somewhere else in the stdlib). Configuration policies are not for us to decide.
On Fri, 13 Mar 2009 11:27:54 am R. David Murray wrote:
Seriously, though, the point is that IMO an application should not be calling fsync unless it provides a way for that behavior to be controlled by the user.
An admirable approach, but also a sweeping generalisation. Who is your expected user-base? Power users, who insist on being given the ability to configure every last aspect of the application behaviour? Or regular users who will be intimidated if you ask them to make the choice? Every configuration choice has a cost: not only does it require more effort to code and maintain, but it leads to a combinatorial explosion of test paths and greater opportunity for bugs. Why pay that cost if your application users won't consider the choice a feature? By all means give the user the option to make that choice, if they will consider it a feature. The point is that these are *application* decisions, not *language* decisions. Python shouldn't be making those decisions, but should be enabling application developers to make them. -- Steven D'Aprano
On Thu, 12 Mar 2009 at 22:09, "Martin v. L�wis" wrote:
Let me try some examples.
Suppose I'm running my applications on a laptop and I don't want the disk to be spinning continually while I work. I'm willing to take the risk of data loss in order to extend my battery life.
So when you select "Save" in your application, would you like the data to be saved, or would you accept that they get lost? If the latter, what kind of interaction would you perform with your application to indicate that you *do* want the data to appear on disk?
I accept that if I have told my laptop to only sync to disk every five minutes (as I have at times done), and it crashes (eg: the battery runs out), then anything I did during those last five minutes will be lost. If the disk then spins up more often than I told it to, I get very annoyed. -- R. David Murray http://www.bitdance.com
On Thu, 2009-03-12 at 20:25 +0000, Antoine Pitrou wrote:
I disagree. The user usually does not know which kind of flushing is needed in order for his data to be safe. Actually, he probably doesn't even know what flushing means, and that files are ever "closed".
However, I also think that any parameter to flush() or close() is a bad idea, since it can't be used when flushing and closing is implicit. For example when the file is used in a "with" statement.
Perhaps this is an argument that the "synciness" of a file should be defined when it is opened? This doesn't give very much control to the programmer, but it certainly seems easy to use correctly. -- Cheers, Leif
On Thu, Mar 12, 2009 at 08:25:59PM +0000, Antoine Pitrou wrote:
However, I also think that any parameter to flush() or close() is a bad idea, since it can't be used when flushing and closing is implicit. For example when the file is used in a "with" statement.
I think the existing os.fsync() and O_SYNC functionality is fine for new applications and packages to write data securely or not. We should just consider whether the stdlib APIs don't make it impossible to write data securely, e.g. dumbdbm's internal file object, and if so, is it worth fixing? --amk
On Fri, 13 Mar 2009 07:25:59 am Antoine Pitrou wrote:
R. David Murray <rdmurray <at> bitdance.com> writes:
By the way, I would not like to see python programmers encouraged to make the same mistake that sqlite3 made. The decision about how aggressive to be on flushing data to disk should be in the hands of the _user_, not the application.
I disagree. The user usually does not know which kind of flushing is needed in order for his data to be safe. Actually, he probably doesn't even know what flushing means, and that files are ever "closed".
Surely it depends on what sort of user you're talking about, and that is often application or OS specific. As a sweeping generalization, Mac users may be more tolerant of slow saves and less tolerant of data loss than Windows users, laptop/notebook users will probably expect the app to honour whatever setting they put in regarding HDD behaviour, and Linux users may expect more fine control over application behaviour and be willing to edit config files to get it. -- Steven D'Aprano
On 3/12/09, "Martin v. Löwis" <martin@v.loewis.de> wrote:
It is starting to look as though flush (and close?) should take an optional wait parameter, to indicate how much re-assurance you're willing to wait for.
Unfortunately, such a thing would be unimplementable on most of today's operating systems.
What am I missing? _file=file class file(_file): ... def flush(self, wait=0): super().flush(self) if wait < 0.25: return if wait < 0.5 and os.fdatasync: os.fdatasync(self.fileno()) return os.fsync(self.fileno()) if wait < 0.75: return if os.ffullsync: os.ffullsync(self.fileno()) (To be honest, I'm not even seeing why it couldn't be done in Objects/fileobject.c, though I realize extension modules would need to go through the python interface to take advantage of it.) -jJ
On Thu, 12 Mar 2009 at 17:01, Jim Jewett wrote:
On 3/12/09, "Martin v. L�wis" <martin@v.loewis.de> wrote:
It is starting to look as though flush (and close?) should take an optional wait parameter, to indicate how much re-assurance you're willing to wait for.
Unfortunately, such a thing would be unimplementable on most of today's operating systems.
What am I missing?
A less confusing name for your proposed parameter :) Maybe 'reliability'?
_file=file class file(_file): ... def flush(self, wait=0): super().flush(self) if wait < 0.25: return if wait < 0.5 and os.fdatasync: os.fdatasync(self.fileno()) return os.fsync(self.fileno()) if wait < 0.75: return if os.ffullsync: os.ffullsync(self.fileno())
(To be honest, I'm not even seeing why it couldn't be done in Objects/fileobject.c, though I realize extension modules would need to go through the python interface to take advantage of it.)
Jim Jewett wrote:
On 3/12/09, "Martin v. Löwis" <martin@v.loewis.de> wrote:
It is starting to look as though flush (and close?) should take an optional wait parameter, to indicate how much re-assurance you're willing to wait for.
Unfortunately, such a thing would be unimplementable on most of today's operating systems.
What am I missing?
As somebody else remarked: I mistook your proposal for a "wait" parameter to denote a time that you want to wait for the data to appear on disk, specified, e.g., in seconds. It didn't occur to me that it might be a unit-less unscaled value, which I find an ugly API. Regards, Martin
Jim Jewett wrote:
On 3/12/09, "Martin v. Löwis" <martin@v.loewis.de> wrote:
It is starting to look as though flush (and close?) should take an optional wait parameter, to indicate how much re-assurance you're willing to wait for.
Unfortunately, such a thing would be unimplementable on most of today's operating systems.
What am I missing?
_file=file class file(_file): ... def flush(self, wait=0): super().flush(self) if wait < 0.25: return if wait < 0.5 and os.fdatasync: os.fdatasync(self.fileno()) return os.fsync(self.fileno()) if wait < 0.75: return if os.ffullsync: os.ffullsync(self.fileno())
What would be wrong with just making the f*sync calls methods of the file object and that's about it? alternatively when flush() should get an optional argument, I'd call it sync and use a set of predefined and meaningful constants (and no floating point value). Just my 2ct. Regards Tino
On Fri, 13 Mar 2009 08:01:27 am Jim Jewett wrote:
On 3/12/09, "Martin v. Löwis" <martin@v.loewis.de> wrote:
It is starting to look as though flush (and close?) should take an optional wait parameter, to indicate how much re-assurance you're willing to wait for.
Unfortunately, such a thing would be unimplementable on most of today's operating systems.
What am I missing?
_file=file class file(_file): ... def flush(self, wait=0): super().flush(self) if wait < 0.25: return if wait < 0.5 and os.fdatasync: os.fdatasync(self.fileno()) return
[snip rest of function] Why are you giving the user the illusion of fine control by making the wait parameter a continuous variable and then using it as if it were a discrete variable? Your example gives only four distinct behaviours, for a (effectively) infinite range of wait. This is bad interface design: it misleads people into thinking that wait=0.4 is 33% safer than wait=0.3 when in fact they are exactly the same. So, replace the wait parameter with a discrete variable -- named or numeric constants. That's a little better, but I still don't think this is the right solution. I believe that we want to leave the foundations as they are now, or at least don't rush into making changes to them. A better approach in my opinion is to leave file as-is (although I wouldn't object much to it growing a sync method, for convenience) and then providing subclasses with the desired behaviour. That scales much better: today we can think of three or four levels of "save reliability" (corresponding to your 0.25, 0.5, 0.7 and 1 values for wait) but next year we might think of six, or ten. Instead of overloading the file type with all these different sorts of behaviour, requiring who knows how many arguments and a complicated API, we leave file nice and simple and allow the application developer to choose the subclass she wants: from filetools import SyncOnWrite as open f = open('mydata.txt', 'w') f.write(data) The choice of which subclass gets used is up to the application, but naturally that might be specified by a user-configurable setting. -- Steven D'Aprano
participants (12)
-
"Martin v. Löwis" -
A.M. Kuchling -
Aahz -
Antoine Pitrou -
Cameron Simpson -
Daniel Stutzbach -
Jim Jewett -
Leif Walsh -
Lie Ryan -
R. David Murray -
Steven D'Aprano -
Tino Wildenhain