using mmap on large (> 2 Gig) files

Travis E. Oliphant oliphant.travis at ieee.org
Tue Oct 24 01:57:31 CEST 2006


Martin v. Löwis wrote:
> myeates at jpl.nasa.gov schrieb:
>> Anyone ever done this? It looks like Python2.4 won't take a length arg
>>> 2 Gig since its not seen as an int. 
> 
> What architecture are you on? On a 32-bit architecture, it's likely
> impossible to map in 2GiB, anyway (since it likely won't fit into the
> available address space).
> 
> On a 64-bit architecture, this is a known limitation of Python 2.4:
> you can't have containers with more than 2Gi items. This limitation
> was removed in Python 2.5, so I recommend to upgrade. Notice that
> the code has seen little testing, due to lack of proper hardware,

NumPy uses the mmap object and I saw a paper at SciPy 2006 that used 
Python 2.5 + mmap + numpy to do some pretty nice and relatively fast 
manipulations of very large data sets.

So, the very useful changes by Martin have seen more testing than he is 
probably aware of.

-Travis




More information about the Python-list mailing list