[BangPypers] Multiplying very large matrices

Anand Balachandran Pillai abpillai at gmail.com
Tue Jan 18 08:50:10 CET 2011


On Sun, Jan 16, 2011 at 12:32 PM, kunal ghosh <kunal.t2 at gmail.com> wrote:

> Hi all,
> I found numpy.memmap
> to be very suitable when matrices larger than the physical memory are
> required.
>
> 1. included in standard numpy installation
> 2. very low learning curve.
>
>
  Interesting package. Just went through the documentation. It is using
  "mmap" system call to map system virtual memory to the process.
  But this approach can also get clumsy if your matrices are
  too huge and you may get into all kinds of paging
  problems if your system doesn't handle virtual memory very well.
  But must be fine on Linux I think.


> pyTables seems to be more suitable but , i somehow found the learning curve
> too steep . Also pyTables needs lot of initializations before anything can
> be done with it
> as compared to memmap.
>
>
  At first look, PyTables might be better than memmap since it uses
 data compression with disk files so you won't hit the memory limit.


> The above reason made me use memmap over pyTable.
>
>


-- 
--Anand


More information about the BangPypers mailing list