Strange performance issue

[Dan Stromberg]

Steven D'Aprano wrote:
> On Mon, 05 Oct 2009 22:31:05 -0700, Dan Stromberg wrote:
>
>   
>> I'm rewriting 3 programs as one program - from Python with Tkinter to
>> Python with pygtk, both on Windows XP.
>>
>> My new version formats an SD card and preallocates some file space in
>> about 3 minutes with "Optimize Performance" selected, and in about 30
>> minutes with "Optimize for Quick Removal" selected.  Needless to say, I
>> don't like the 27 minute penalty much.
>>     
>
> I don't understand what that means. How are you formatting the SD card?
>   
The FormatEx function via ctypes.
> I'm guessing that Optimize for Quick Removal means that every write is 
> immediately synced to disk. That will probably be slow, no matter what.
>   
Yes, presumably Optimize for Quick Removal is a write-through cache 
(synchronous) and "Optimize for Performance" is a write-back cache 
(asynchronous).
>   
>> But the old version of the program formats and preallocates in 3 minutes
>> irrespective of whether the drive is in "optimize performance" or
>> "optimize for quick removal".
>>     
>
> I suspect that if there was no performance penalty in the old version, it 
> was because you inadvertently were always using "Optimize Performance" no 
> matter what.
>   
I'm pretty confident that unless Windows was lying to me, that I did 
some tests with Optimize for Performance and some tests without.
> BTW, if you want to pre-allocate some space, and you don't care what is 
> in the file, you *may* be able to use file.truncate() to do so quickly. 
> Despite the name, it doesn't just truncate files, it can also be used to 
> extend them. (At least on POSIX systems.)
>   
On a POSIX system, I'd normally just seek and write a single null to get 
a file with holes.  But this is not only Windows, but FAT32 - and the 
data will later be read without going through the filesystem.  It seems 
prudent to actually write the blocks this time.


Thanks, and see my other message (to be sent momentarily) for an 
apparent solution.
>>>> f = open('empty', 'w')
>>>> f.seek(1000)
>>>> f.truncate()
>>>> f.close()
>>>> f = open('empty', 'r')
>>>> len(f.read())
>>>>         
> 1000
>   
>>>> f.close()
>>>>         
>
>
>
>