[Tutor] Read-ahead for large fixed-width binary files?

Marc Tompkins marc.tompkins at gmail.com
Fri Nov 16 20:10:14 CET 2007

I've been writing a lot of utility programs for my clients who are
users of a certain legacy database application - exporting, reporting,
pretty label printing, etc.  The database is normalized into 45 files,
each of which contains fixed-length records (ranging from 80 bytes to
1024).  A few of those files contain relatively few records, while
others - depending how long the users have worked with the application
- may contain millions of records; the transaction file is generally
the largest and consists of 256-byte records.

(Before anybody asks, yes!  I know that the best way to do this would
be to use the database engine that created the files.  Unfortunately,
that's not really an option.)
I am NOT writing back to the data file.

I seem to have two alternatives - read the file one record at a time
(which means I spend a ridiculous amount of time waiting for the
disk), or read the whole thing at once and process it in memory
(which, depending on the user's machine, will probably choke on a
250MB transaction file.)

My question is this: does anybody know of an equivalent to
"readlines(sizehint)" for non-delimited, binary files?  I've Googled
and Googled until I'm groggy, but I don't seem to find what I want.



More information about the Tutor mailing list