[Tutor] Read-ahead for large fixed-width binary files?
marc.tompkins at gmail.com
Fri Nov 16 20:15:40 CET 2007
OK, I feel quite foolish... almost immediately after hitting 'send' I
realized I can implement this myself, using 'read(bigsize)' -
currently I'm using 'read(recordsize)'; I just need to add an extra
loop around my record reads. Please disregard...
On Nov 16, 2007 11:10 AM, Marc Tompkins <marc.tompkins at gmail.com> wrote:
> I've been writing a lot of utility programs for my clients who are
> users of a certain legacy database application - exporting, reporting,
> pretty label printing, etc. The database is normalized into 45 files,
> each of which contains fixed-length records (ranging from 80 bytes to
> 1024). A few of those files contain relatively few records, while
> others - depending how long the users have worked with the application
> - may contain millions of records; the transaction file is generally
> the largest and consists of 256-byte records.
> (Before anybody asks, yes! I know that the best way to do this would
> be to use the database engine that created the files. Unfortunately,
> that's not really an option.)
> I am NOT writing back to the data file.
> I seem to have two alternatives - read the file one record at a time
> (which means I spend a ridiculous amount of time waiting for the
> disk), or read the whole thing at once and process it in memory
> (which, depending on the user's machine, will probably choke on a
> 250MB transaction file.)
> My question is this: does anybody know of an equivalent to
> "readlines(sizehint)" for non-delimited, binary files? I've Googled
> and Googled until I'm groggy, but I don't seem to find what I want.
More information about the Tutor