[BangPypers] Multiplying very large matrices

[kunal ghosh]

Hi all,
I found numpy.memmap
to be very suitable when matrices larger than the physical memory are
required.

1. included in standard numpy installation
2. very low learning curve.

pyTables seems to be more suitable but , i somehow found the learning curve
too steep . Also pyTables needs lot of initializations before anything can
be done with it
as compared to memmap.

The above reason made me use memmap over pyTable.

On Sat, Jan 15, 2011 at 11:34 PM, kunal ghosh <kunal.t2 at gmail.com> wrote:

> Thanks Santosh ,
>
> This stack overflow thread indeed discusses the exact same problem i have.
> Wonder how i missed it :) in my preliminary searches.
>
> thanks again !
>
>
> On Sat, Jan 15, 2011 at 11:12 PM, Santosh Rajan <santrajan at gmail.com>wrote:
>
>> Hope this helps
>>
>> http://stackoverflow.com/questions/1053928/python-numpy-very-large-matrices
>>
>> On Sat, Jan 15, 2011 at 10:11 PM, kunal ghosh <kunal.t2 at gmail.com> wrote:
>> > Hi all,
>> > while implementing Locality Preserving Projections ,
>> > at one point i have to perform X L X.transpose()
>> > these matrices are large (32256 x 32256) so i get "out of memory" error.
>> >
>> > I assume, as the dataset gets larger one would come across this problem
>> ,
>> > how would
>> > one go about solving this ? Is there a common trick that is used to deal
>> > with such problems ?
>> > Or the workstation calculating these problems needs to have HUGE
>>  amounts of
>> > physical memory ?
>> >
>> > I am using python and numpy / scipy
>> >
>> > --
>> > regards
>> > -------
>> > Kunal Ghosh
>> > Dept of Computer Sc. & Engineering.
>> > Sir MVIT
>> > Bangalore,India
>> >
>> > permalink: member.acm.org/~kunal.t2
>> > Blog:kunalghosh.wordpress.com
>> > Website:www.kunalghosh.net46.net
>> > _______________________________________________
>> > BangPypers mailing list
>> > BangPypers at python.org
>> > http://mail.python.org/mailman/listinfo/bangpypers
>> >
>>
>>
>>
>> --
>> http://about.me/santosh.rajan
>> _______________________________________________
>> BangPypers mailing list
>> BangPypers at python.org
>> http://mail.python.org/mailman/listinfo/bangpypers
>>
>
>
>
> --
> regards
> -------
> Kunal Ghosh
> Dept of Computer Sc. & Engineering.
> Sir MVIT
> Bangalore,India
>
> permalink: member.acm.org/~kunal.t2
> Blog:kunalghosh.wordpress.com
> Website:www.kunalghosh.net46.net
>
>
>


-- 
regards
-------
Kunal Ghosh
Dept of Computer Sc. & Engineering.
Sir MVIT
Bangalore,India

permalink: member.acm.org/~kunal.t2
Blog:kunalghosh.wordpress.com
Website:www.kunalghosh.net46.net