[Numpy-discussion] numpy videos

[Sturla Molden]

On 12.03.2012 23:23, Abhishek Pratap wrote:
> Super awesome. I love how the python community in general keeps the
> recordings available for free.
>
> @Adam : I do have some problems that I can hit numpy with, mainly
> bigData based. So in summary I have millions/billions of rows of
> biological data on which I want to run some computation but at the
> same time have a capability to do quick lookup. I am not sure if numpy
> will be applicable for quick lookups  by a string based key right ??


Jason Kinser's book on Python for bioinformatics might be of interest. 
Though I don't always agree with his NumPy coding style.

As for "big data", it is a problem regardless of language. The HDF5 
library might be of help (cf. PyTables or h5py, I actually prefer the 
latter).

With a 64 bit system it is also possible to memory map a temporary file, 
and tell the OS to keep as much of it in memory if possible. That way we 
can "fake" more RAM than we actually have. (The Linux equivalent of the 
code in bigmem.c would be to mmap from tmpfs.) A usecase for bigmem.c is 
e.g. if you need to use 10 tables that each are 1-2 GB in size, but only 
have 4 GB of RAM on the desktop computer.


Sturla










-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: bigmem.c
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20120313/09ae1b17/attachment.c>