Manipulate Large Binary Files

Derek Martin code at
Wed Apr 2 11:50:51 EDT 2008

On Wed, Apr 02, 2008 at 10:59:57AM -0400, Derek Tracy wrote:
> I generated code that works wonderfully for files under 2Gb in size
> but the majority of the files I am dealing with are over the 2Gb
> limit
> ary = array.array('H',

You're trying to read the file all at once.  You need to break your
reads up into smaller chunks, in a loop.  You're essentially trying to
store more data in memory than your OS can actually access in a single

Something like this (off the top of my head, I may have overlooked
some detail, but it should at least illustrate the idea):

# read a meg at a time
buffsize = 1048576
while true:
	buff =
	if len(buff) != buffsize:

Derek D. Martin
GPG Key ID: 0x81CFE75D

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <>

More information about the Python-list mailing list