using mmap on large (> 2 Gig) files
timr at probo.com
Thu Oct 26 06:28:21 CEST 2006
"sturlamolden" <sturlamolden at yahoo.no> wrote:
>However, "memory mapping" a file by means of fseek() is probably more
>efficient than using UNIX' mmap() or Windows'
My goodness, do I disagree with that! At least on Windows, I/O on a file
mapped with MapViewOfFile uses the virtual memory pager -- the same
mechanism used by the swap file. Because it is so heavily used, that is
some of the most well-optimized code in the system.
>We can implement a container object backed by a binary file just as
>efficient (and possibly even more efficient) without using the OS'
>memory mapping facilities. The major advantage is that we can
>"pseudo-memory map" a lot more than a 32 bit address space can harbour.
Both the Unix mmap and the Win32 MapViewOfFile allow a starting byte
offset. It wouldn't be rocket science to extend Python's mmap to allow
>There are in any case room for improving Python's mmap object.
Here we agree.
Tim Roberts, timr at probo.com
Providenza & Boekelheide, Inc.
More information about the Python-list