Implementing file reading in C/Python
invalid at invalid
Mon Jan 12 16:43:06 CET 2009
On 2009-01-12, Sion Arrowsmith <siona at chiark.greenend.org.uk> wrote:
> Grant Edwards <invalid at invalid> wrote:
>>On 2009-01-09, Sion Arrowsmith <siona at chiark.greenend.org.uk> wrote:
>>> Grant Edwards <invalid at invalid> wrote:
>>>>If I were you, I'd try mmap()ing the file instead of reading it
>>>>into string objects one chunk at a time.
>>> You've snipped the bit further on in that sentence where the
>>> OP says that the file of interest is 2GB. Do you still want to
>>> try mmap'ing it?
>>Sure. The larger the file, the more you gain from mmap'ing it.
>>2GB should easily fit within the process's virtual memory
> Assuming you're in a 64bit world. Me, I've only got 2GB of address
> space available to play in -- mmap'ing all of it out of the question.
Oh. I assumed that decent 32-bit OSes would provide at least 3-4GB
of address space to user processes. What OS are you using?
> But I supposed that mmap'ing it chunk at a time instead of
> reading chunk at a time might be worth considering.
I'd try mmap'ing it in large chunks (512MB maybe).
Grant Edwards grante Yow! I feel like a wet
at parking meter on Darvon!
More information about the Python-list