[Tutor] Read-ahead for large fixed-width binary files?
marc.tompkins at gmail.com
Fri Nov 16 20:10:14 CET 2007
I've been writing a lot of utility programs for my clients who are
users of a certain legacy database application - exporting, reporting,
pretty label printing, etc. The database is normalized into 45 files,
each of which contains fixed-length records (ranging from 80 bytes to
1024). A few of those files contain relatively few records, while
others - depending how long the users have worked with the application
- may contain millions of records; the transaction file is generally
the largest and consists of 256-byte records.
(Before anybody asks, yes! I know that the best way to do this would
be to use the database engine that created the files. Unfortunately,
that's not really an option.)
I am NOT writing back to the data file.
I seem to have two alternatives - read the file one record at a time
(which means I spend a ridiculous amount of time waiting for the
disk), or read the whole thing at once and process it in memory
(which, depending on the user's machine, will probably choke on a
250MB transaction file.)
My question is this: does anybody know of an equivalent to
"readlines(sizehint)" for non-delimited, binary files? I've Googled
and Googled until I'm groggy, but I don't seem to find what I want.
More information about the Tutor