[Numpy-discussion] Not enough storage for memmap on 32 bit Win XP for accumulated file size above approx. 1 GB

[Kim Hansen]

OS. Win XP SP3, 32 bits
Python: 2.5.4
Numpy: 1.3.0

I have am having some major problems converting a 750 MB recarray into
a 850 MB recarray

To save RAM I would like to use a read-only and a writeable memap for
the two recarrays during the conversion.

So I do something like:

import os
from stat import ST_SIZE
import numpy as np
...
records = os.stat(toconvert_path)[ST_SIZE] / toconvert_dtype.itemsize
toconvert = np.memmap(toconvert_path, dtype=toconvert_dtype,
mode="r").view(np.recarray)
result = np.memmap(result_path, dtype = result_dtype, mode = "w+",
shape=(records,))

The code manages to create the toconvert memmap (750 MB),
but when trying to create the second memap object I get
  File "C:\Python25\Lib\site-packages\numpy\core\memmap.py", line 226,
in __new__
    mm = mmap.mmap(fid.fileno(), bytes, access=acc)
WindowsError: [Error 8] Not enough storage is available to process this command

By tracing before and after, I can see the file size is zero before
calling mmap.mmap and has the expected 850 MB size after the
WindowsError has been thrown somewhere inside mmap.mmap. There is 26
GB of free disc space, so the error message seems wrong?

If I comment out the creation of the first memmap, I can successfully
create the result memmap, so the error seems to be related to the
accumulated size of all mmap.mmaps generated. I have other cases with
somewhat smaller files to convert where the transition is OK. It seem
like I begin to get these problems when the accumulated size of
memmaps exceeds 1GB

I am surprised by this, as

http://docs.scipy.org/doc/numpy/reference/arrays.classes.html#memory-mapped-file-arrays

mentions there are upper bounds to the size when using versions of
python before 2.5.

>From this I had the impression that there was no size limit as long as
you were using ver. >=2.5 (as I am)

Is it due to the 32 bit OS I am using?

Is there anything I can do to resolve the problem?

Best wishes,

Kim