2 GB limit for memmap'ed files
A long time ago, Travis wrote:
On a related, but orthogonal note:
My understanding is that using memory-mapped files for *very* large files will require modification to the mmap module in Python --- something I think we should push. One part of that process would be to add the C-struct array interface to the mmap module and the buffer object -- perhaps this is how we get the array interface into Python quickly. Then, if we could make a base-type mmap that did not use the buffer interface or the sequence interface (similar to the bigndarray in scipy_core) and therefore by-passed the problems with Python in those areas, then the current mmap object could inherit from the base class and provide current functionality while still exposing the array interface for access to >2GB files on 64-bit systems.
Who would like to take up the ball for modifying mmap in Python in this fashion?
-Travis
Did anyone ever "pick up the ball" on this issue? Glen
On 9/7/06, Glen W. Mabey
A long time ago, Travis wrote:
My understanding is that using memory-mapped files for *very* large files will require modification to the mmap module in Python ---
Did anyone ever "pick up the ball" on this issue?
This works with python-2.5 betas and above, and numpy 1.0 betas and above. I frequently do 10+ GB mmaps. Mike -- mike.ressler@alum.mit.edu
Hi Glen ! How is that quote really !? The new Python2.5 *is* implementing the needed changes - so go ahead install Python2.5 (rc1 is the latest I think) and report how it works. I would also be very intersted to hear ;-) -Sebastian Haase On Thursday 07 September 2006 12:34, Glen W. Mabey wrote:
A long time ago, Travis wrote:
On a related, but orthogonal note:
My understanding is that using memory-mapped files for *very* large files will require modification to the mmap module in Python --- something I think we should push. One part of that process would be to add the C-struct array interface to the mmap module and the buffer object -- perhaps this is how we get the array interface into Python quickly. Then, if we could make a base-type mmap that did not use the buffer interface or the sequence interface (similar to the bigndarray in scipy_core) and therefore by-passed the problems with Python in those areas, then the current mmap object could inherit from the base class and provide current functionality while still exposing the array interface for access to >2GB files on 64-bit systems.
Who would like to take up the ball for modifying mmap in Python in this fashion?
-Travis
Did anyone ever "pick up the ball" on this issue?
Glen
------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
participants (3)
-
Glen W. Mabey -
Mike Ressler -
Sebastian Haase