Swap memory in Python ? - three questions
Marc Christiansen
usenet at solar-empire.de
Tue Jul 29 09:26:55 EDT 2008
Robert LaMarca <robertmlamarca at yahoo.com> wrote:
> Hi,
>
> I am using numpy and wish to create very large arrays. My system is
> AMD 64 x 2 Ubuntu 8.04. Ubuntu should be 64 bit. I have 3gb RAM and a
> 15 GB swap drive.
>
> The command I have been trying to use is;
> g=numpy.ones([1000,1000,1000],numpy.int32)
>
> This returns a memory error.
Works for me on AMD64x2, 2GB RAM, 3GB swap. With much paging of course
;) Process size of python 4019508kB.
> A smaller array ([500,500,500]) worked fine..
About 0.5GB.. no surprise.
> Two smaller arrays again crashed the system.
Crash as in "computer reboots"? Strange. And you say, that two [500,
500, 500] arrays are two much?
> So... I did the math. a 1000x1000x1000 array at 32 bits should be
> around 4gb RAM... Obviously larger than RAM, but much smaller than the
> swap drive.
Sounds like either your kernel or python are not 64bit.
For the kernel, have a look at /proc/meminfo, which value you get for
VmallocTotal. I have VmallocTotal: 34359738367 kB.
For python, check sys.maxint. If it's 2147483647, then you have a 32bit
python. Mine says 9223372036854775807.
> 1. So... does Numpy have a really lot of overhead? Or is my system
> just not somehow getting to make use of the 15gb swap area.
No. Yes.
> 2. Is there a way I can access the swap area, or direct numpy to do
> so? Or do I have to write out my own numpy cache system...
> 3. How difficult is it to use data compression internally on numpy
> arrays?
2 + 3: Should not be necessary.
I just tried a 32bit python on my 64bit system, using Numeric instead of
numpy (don't have a 32bit numpy ready):
g=Numeric.zeros([1000,1000,1000],Numeric.Int32)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError: can't allocate memory for array
g=Numeric.zeros([1000,1000,500],Numeric.Int32) succeeds.
So it looks like you have a 32bit kernel.
Marc
More information about the Python-list
mailing list