Manipulate Large Binary Files

[Derek Tracy]

---------------------------
Derek Tracy
tracyde at gmail.com
---------------------------

On Apr 3, 2008, at 3:03 AM, Paul Rubin <"http:// 
phr.cx"@NOSPAM.invalid> wrote:

> Derek Martin <code at pizzashack.org> writes:
>>> Both are clocking in at the same time (1m 5sec for 2.6Gb), are there
>>> any ways I can optimize either solution?
>
> Getting 40+ MB/sec through a file system is pretty impressive.
> Sounds like a RAID?
>
>> That said, due to normal I/O generally involving double-buffering,  
>> you
>> might be able to speed things up noticably by using Memory-Mapped I/O
>> (MMIO).  It depends on whether or not the implementation of the  
>> Python
>> things you're using already use MMIO under the hood, and whether or
>> not MMIO happens to be broken in your OS. :)
>
> Python has the mmap module and I use it sometimes, but it's not
> necessarily the right thing for something like this.  Each page you
> try to read from results in own delay while the resulting page fault
> is serviced, so any overlapped i/o you get comes from the OS being
> nice enough to do some predictive readahead for you on sequential
> access if it does that.  By coincidence there are a couple other
> threads mentioning AIO which is a somewhat more powerful mechanism.
>
> -- 
> http://mail.python.org/mailman/listinfo/python-list