[BangPypers] Multiplying very large matrices
Anand Balachandran Pillai
abpillai at gmail.com
Tue Jan 18 08:50:10 CET 2011
On Sun, Jan 16, 2011 at 12:32 PM, kunal ghosh <kunal.t2 at gmail.com> wrote:
> Hi all,
> I found numpy.memmap
> to be very suitable when matrices larger than the physical memory are
> 1. included in standard numpy installation
> 2. very low learning curve.
Interesting package. Just went through the documentation. It is using
"mmap" system call to map system virtual memory to the process.
But this approach can also get clumsy if your matrices are
too huge and you may get into all kinds of paging
problems if your system doesn't handle virtual memory very well.
But must be fine on Linux I think.
> pyTables seems to be more suitable but , i somehow found the learning curve
> too steep . Also pyTables needs lot of initializations before anything can
> be done with it
> as compared to memmap.
At first look, PyTables might be better than memmap since it uses
data compression with disk files so you won't hit the memory limit.
> The above reason made me use memmap over pyTable.
More information about the BangPypers