File IO question.

J Poirier oo1z at
Thu Jul 15 22:26:57 CEST 2004

On Thu, 15 Jul 2004 12:38:36 -0500, Larry Bates
<lbates at> wrote:

> You answered your own question:
>> I'm guessing the hard drives would fill up
>> anyway regardless of the size of the hard drive,
>> crashes just wouldn't occur quite as often.
> If this statement is true, compressing the files
> help either as that is just the same as a larger
> drive.
> But, you could limit the size and number of trace
> and periodically throw away the oldest ones to keep
> trace files at a nearly constant size.  I just can't
> from your description exactly what is getting
written to
> the trace file that makes it so large (debugging
> Maybe you could only log errors/warnings to the
> file so it grows more slowly?
> Larry Bates
> Syscon, Inc.
> "J Poirier" <oo1z at> wrote in message
news:mailman.445.1089911565.5135.python-list at
>> Hi All,
>> I'm hoping that someone might have some pointers or
>> examples on how to proceed with a solution to the
>> following problem:
>> A test application, which produces a trace file, is
>> being run for very long periods of time. Say 72
>> or more.
>> The application is often running on older PCs that
>> have relatively small hard drives in comparison to
>> to how big the trace file can become.
>> The trace files tend to accumulate as they're not
>> always deleted once archived to a network database,
>> and often times a machine will crash in the middle
>> a
>> very long test due to the hard drive filling up.
>> There are dozens of PCs being used for the tests so
>> buying bigger hard drives isn't really feasible.
>> I'm guessing the hard drives would fill up
>> anyway regardless of the size of the hard drive,
>> crashes just wouldn't occur quite as often.
>> The nice thing is that the trace files compress
>> well.
>> I messed around with the mmap and file object stuff
>> as well as the win32 extensions thinking that I
>> extract and compress the data that was being
>> to
>> the trace file, by the application, in chuncks.
>> Although I was able to get it to work on a
>> setup, it didn't work when used with the real
>> application.
>> Any hints on how to get something similar to the
>> to work or recommendations on alternate solutions
>> would
>> be *greatly* appreciated.
>> Thanks,
>> Joe

The files contain lots of detailed mobile info
(RLC/MAC, etc) from GSM and WCDMA networks.

Running a test involves running through a bunch of

All test cases must finish without interruption
for the trace data to be considered valid. If an
interruption does occur the entire test must be 
re-started from the beginning. 

It's easy to have a trace that's several hundred meg
several gig for a test that's being run for several

A trace file can usually be compressed to a tenth of
its original uncompressed size. So removing the info
from a trace file that the application is writing to,
while it's still periodically being written to by the
test application, and compressing the piped out trace
data, seemed like the way to go. 

I was unable to retrieve any of the trace data from
a file that was created by the application, and the
application wouldn't write any trace info to a file
created by my script, while the script was accessing
the file.

Ultimately at some point old trace files need to be
deleted from the hard drive, but that doesn't help 
with handling the size of the trace file. 

If a file's data could be piped to a compressed file
as the trace was being taken, more comprehensive test
cases could be run at a single session.

E.g., a 7 day test would currently be out of the
question because it generates a 25 gig trace. But if
the trace file's contents was piped to a compressed 
file it would only end up taking up 2.5 gig of space
once the test completed.

Unfortunately my script was unable to process a trace
file created by the application and the application
wouldn't write to a file that my script created while
the script was accessing the file.


Do you Yahoo!?
Read only the mail you want - Yahoo! Mail SpamGuard. 

More information about the Python-list mailing list