[SciPy-User] Multiplying very large matrices

[Gael Varoquaux]

On Sat, Jan 15, 2011 at 10:09:59PM +0530, kunal ghosh wrote:
>    while implementing Locality Preserving Projections ,
>    at one point i have to perform X L X.transpose()
>    these matrices are large (32256 x 32256) so i get "out of memory" error.
>    I assume, as the dataset gets larger one would come across this problem ,
>    how would
>    one go about solving this ? Is there a common trick that is used to deal
>    with such problems ?
>    Or the workstation calculating these problems needs to have HUGE  amounts
>    of physical memory ?

Maybe there is a random projection/random sampling algorithm to solve
your problem on partial data:

http://metaoptimize.com/qa/questions/1640/whats-a-good-introduction-to-random-projections

Gael