Memory Error while simulating matrix

[David C. Ullrich]

On Tue, 26 Mar 2002 05:08:49 -0600, Skip Montanaro <skip at pobox.com>
wrote:

>
>    slick> When i execute the code(part of an algorithm code) given below, i
>    slick> get a MemoryError.
>
>    row = []                # list 
>    row = [0] * 1000000     # each row contains million columns 
>    matrix =[]              # contains a collection of rows  
>
>    for i in range(1000):   # creating a matix of 1000 rows each 
>      matrix.append(row[:]) # having a million  columns
>
>You are trying to create 1,000 lists, each with 1,000,000 elements, or 10**9
>4-byte object pointers, so at minimum you will be chewing 4GB of virtual
>memory, probably much more once your matrix becomes a bit less uniform.  To
>keep from getting a MemoryError you should have at least that much swap
>space.  Actually much more. 

??? The memory in the swap file is still part of the virtual 32-bit
memory space, right? I don't see how a person could use more than
4GB of memory, regardless of whether it's virtual or not, on a
32-bit system. (Or 2GB on Windows, cuz it wants to reserve 2GB
for its own purposes...)

> Integer overhead is 12 bytes per object.
>(Float overhead is 16 bytes per object.)  Assuming most of the elements of
>your matrix are not recycled integers between -1 and 100, your memory
>consumption will be closer to 20GB than 4GB.
>
>Obviously, you can use the array module or NumPy.  Both of those are going
>to push the memory consumption closer to the 4GB end of things.
>
>-- 
>Skip Montanaro (skip at pobox.com - http://www.mojam.com/)
>


David C. Ullrich