writing large files quickly

Fredrik Lundh fredrik at pythonware.com
Sun Jan 29 09:59:35 CET 2006

Bengt Richter wrote:

> >> How the heck does that make a 400 MB file that fast? It literally takes
> >> a second or two while every other solution takes at least 2 - 5 minutes.
> >> Awesome... thanks for the tip!!!
> >
> >Because it isn't really writing the zeros.   You can make these
> >files all day long and not run out of disk space, because this
> >kind of file doesn't take very many blocks.   The blocks that
> >were never written are virtual blocks, inasmuch as read() at
> >that location will cause the filesystem to return a block of NULs.
> >
> I wonder if it will also "write" virtual blocks when it gets real
> zero blocks to write from a user, or even with file system copy utils?

I've seen this behaviour on "big iron" Unix systems, in a benchmark that
repeatedly copied data from a memory mapped section to an output file.

but for the general case, I doubt that adding "is this block all zeros" or
"does this block match something we recently wrote to disk" checks will
speed things up, on average...


More information about the Python-list mailing list