[Numpy-discussion] Not enough storage for memmap on 32 bit WinXP for accumulated file size above approx. 1 GB

[Kim Hansen]

2009/7/24 Citi, Luca <lciti at essex.ac.uk>:
> Hello!
> I have access to both a 32bit and a 64bit linux machine.
>
> I had to change your code (appended) because I got an error about
> not being able to create a mmap larger than the file.
> Here are the results...
>
> On the 32bit machine:
>
> lciti at xps2:~$ python /tmp/ppp.py
> Created 1 writeable mmaps containing 100 MB
> Created 2 writeable mmaps containing 200 MB
> Created 3 writeable mmaps containing 300 MB
> Created 4 writeable mmaps containing 400 MB
> Created 5 writeable mmaps containing 500 MB
> [......]
> Created 24 writeable mmaps containing 2400 MB
> Created 25 writeable mmaps containing 2500 MB
> Created 26 writeable mmaps containing 2600 MB
> Created 27 writeable mmaps containing 2700 MB
> Created 28 writeable mmaps containing 2800 MB
> Created 29 writeable mmaps containing 2900 MB
> Created 30 writeable mmaps containing 3000 MB
> Removing mmaps...
> Done...
> Traceback (most recent call last):
>  File "/tmp/ppp.py", line 19, in <module>
>    mm = mmap.mmap(f.fileno(), 0)
> mmap.error: [Errno 12] Cannot allocate memory
>
>
>
>
> On the 64bit machine I can create 510 mmaps
> both with bytes_per_mmap at 100MiB and 1GiB:
>
> Created 1 writeable mmaps containing 1000 MB
> Created 2 writeable mmaps containing 2000 MB
> Created 3 writeable mmaps containing 3000 MB
> Created 4 writeable mmaps containing 4000 MB
> Created 5 writeable mmaps containing 5000 MB
> Created 6 writeable mmaps containing 6000 MB
> [......]
> Created 501 writeable mmaps containing 501000 MB
> Created 502 writeable mmaps containing 502000 MB
> Created 503 writeable mmaps containing 503000 MB
> Created 504 writeable mmaps containing 504000 MB
> Created 505 writeable mmaps containing 505000 MB
> Created 506 writeable mmaps containing 506000 MB
> Created 507 writeable mmaps containing 507000 MB
> Created 508 writeable mmaps containing 508000 MB
> Created 509 writeable mmaps containing 509000 MB
> Created 510 writeable mmaps containing 510000 MB
> Removing mmaps...
> Done...
> Traceback (most recent call last):
>  File "/tmp/ppp.py", line 19, in <module>
>    mm = mmap.mmap(f.fileno(), 0)
> mmap.error: [Errno 24] Too many open files
>
>
>
> I do not even have 510GiB free in the disk. But I
> think that is because the ext3 filesystem allows
> sparse files.
>
> I think this shows that the maximum mapped space
> cannot be more than the maximum address space
> which is 2**64 for 64bit machines, 2GiB for windows32
> and 3GiB for linux32.
>
> Under WindowsXP, you can try to increase it from 2GiB to
> 3GiB using the /3GB switch in the boot.ini
>
> Best,
> Luca
>
Hi Luca, thx for trying. Your test clearly shows that 32 bits imposes
a severe limitation.

I tried adding the /3GB switch to boot.ini as you suggested:
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP
Professional" /noexecute=optin /fastdetect /3GB
and rebooted the system.

Unfortunately that did not change anything for me. I still hit a hard
deck around 1.9 GB. Strange.

Best wishes,
Kim