How to safely maintain a status file
steve+comp.lang.python at pearwood.info
Fri Jul 13 09:14:36 CEST 2012
On Thu, 12 Jul 2012 21:26:20 -0700, rantingrickjohnson wrote:
> On Thursday, July 12, 2012 10:13:47 PM UTC-5, Steven D'Aprano wrote:
>> Rick has obviously never tried to open a file for reading when somebody
>> else has it opened, also for reading, and discovered that despite
>> Windows being allegedly a multi-user operating system, you can't
>> actually have multiple users read the same files at the same time.
> You misread my response. My comment was direct result of Christian
> (paraphrase) "On some systems you are not permitted to delete a file
> whilst the file is open "
> ...which seems to be consistent to me. Why would *anybody* want to
> delete a file whilst the file is open?
Because it is useful and a sensible thing to do.
Why should one misbehaved application, keeping a file open, be allowed to
hold every other application, and the file system, hostage?
This is one of the many poor decisions which makes Windows so vulnerable
to viruses and malware. If malware can arrange to keep itself open, you
can't delete it. Thanks guys!
> Bringing back the car analogy
> again: Would you consider jumping from a moving vehicle a consistent
> interaction with the interface of a vehicle? Of course not. The
> interface for a vehicle is simple and consistent:
> 1. You enter the vehicle at location A
> 2. The vehicle transports you to location B
> 3. You exit the vehicle
Amusingly, you neglected to specify "the vehicle stops" -- and rightly
so, because of course having to stop the vehicle is not a *necessary*
condition for exiting it, as tens of thousands of stunt men and women can
Not to mention people parachuting out of an airplane, pirates or
commandos boarding a moving ship, pedestrians transferring from a slow
moving walkway to a faster moving walkway, farmers jumping off a trailer
while it is still being towed behind a tractor (and jumping back on
again), and Bruce Willis in "Red" in very possibly the best slow-motion
action sequence in the history of Hollywood.
> At no time during the trip would anyone expect you to leap from the
Expected or not, you can do so.
> But when you delete open files, you are essentially leaping
> from the moving vehicle! This behavior goes against all expectations of
> consistency in an API -- and against all sanity when riding in a
Fortunately, files on a file system are not cars, and deleting open files
is a perfectly reasonable thing to do, no more frightening than in Python
deleting a reference to an object using the del statement. Imagine how
stupid it would be if this happened:
py> x = 42
py> y = x
py> del y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
DeleteError: cannot delete reference to object '42' until no other
references to it exist
Fortunately, Python doesn't do that -- it tracks when the object is no
longer being accessed, and only then physically reclaims the memory used.
And so it is on POSIX file systems: the file system keeps track of when
the file on disk is no longer being accessed, and only then physically
reclaims the blocks being used. Until then, deleting the file merely
unlinks the file name from the blocks on disk, in the same way that
"del y" merely unlinks the name y from the object 42.
>> Opening files for exclusive read *by default* is a pointless and silly
>> limitation. It's also unsafe: if a process opens a file for
>> exclusive read, and then dies, *no other process* can close that file.
> Oh come on. Are you actually going to use "errors" or "unintended
> consequences", or even "Acts of God" to defend your argument?
Features have to be judged by their actual consequences, not some
unrealistic sense of theoretical purity. The actual consequences of
mandatory exclusive file locking is, *it sucks*.
Windows users are used to having to reboot their server every few days
because something is broken, so they might not mind rebooting it because
some file is locked in a mandatory open state and not even the operating
system can unlock it. But for those with proper operating systems who
expect months of uninterrupted service, mandatory locking is a problem to
be avoided, not a feature.
> Okay. I suppose "IF" the car spontaneously combusted "THEN" the
> passengers would be wise to jump out, leaving the vehicle to the whims
> of inertia.
In this analogy, is the car the file name, the inode, or the directory?
Are the passengers the file name(s), or the file contents, or the inode?
Is the driver meant to be the file system? If I have a hard link to the
file, does that mean the passengers are in two cars at once, or two lots
of passengers in the same car?
>> One neat trick is to open a file, then delete it from disk while it is
>> still open. So long as your process is still running, you can write to
>> this ghost file, as normal, but no other process can (easily) see it.
>> And when your process ends, the file contents is automatically deleted.
> Well "neat tricks" aside, I am of the firm belief that deleting files
> should never be possible whilst they are open.
[condescension = ON]
Good for you Rick. Having strongly held opinions on things you have only
a limited understanding about is your right as an American.
[condescension = OFF]
> * Opening files requires that data exist on disk
> * Reading and writing files requires an open file obj
You have missed a step in jumping from files on disk to open file objects.
Open file objects do not necessarily correspond to files on disk. For
example, in standard Pascal, file objects are purely in-memory constructs
emulating files on a tape drive, with no relationship to on-disk files.
(Any half-decent Pascal compiler or interpreter will *also* give you ways
to access real files on disk, but that isn't covered by the standard.)
Even when the file object does come from an actual disk file, we can
conclude that before you can open a file for reading, it must exist; but
having opened it, there is no *necessary* requirement that it *remains*
on disk. If you try to read from an open file object whose underlying
file has been deleted, there are three perfectly reasonable behaviours:
- you get an error;
- it is the same result as if the file has been truncated to zero bytes;
- deleting the file only deletes the *name*, not contents, until the
last open file handle is shut, and then the contents are deleted.
> * Closing files requires an open file object
Naturally; but open file objects don't require that the on-disk file
> * And deleting files requires that the file NOT be open
Not at all.
> Would you also entertain the idea of reading or writing files that do
> not exist? (not including pseudo file objs like StringIO of course!).
Define "file" and "exist". Because you are conflating at least three
a file name
an inode (blocks on a disk)
a file object
Of course it is useful to break the abstraction that file objects must be
files on disk. StringIO is one such example. Standard Pascal file objects
is another. Likewise, it is useful to be able to read from an inode that
is no longer connected to a file name.
So, absolutely, yes, it is useful to be able to read and write from files
that don't exist, under some circumstances.
> Summary: Neat tricks and Easter eggs are real hoot, but consistency
> in APIs is the key.
A foolish consistency is the hobgoblin of little minds.
More information about the Python-list