[Numpy-discussion] numpy videos

[Adam Hughes]

This is a probably an area that is quite common, so I'd be interested to
hear some other chime in.  I refer to the lookup and storage in numpy data.

Your implementation will of course be unique, but there are several avenues
that you can consider.  Here is how I handle a similar problem.

Imagine I have data, probably similar to yours, where there is qualitative
data (maybe biological or experimental parameters and other things), as
well as numerical data.  I would define a dictionary object that stores
both of these to a unique key.  In my work, I use the original file that
all the information was taken from as my key.  So for example:

dict{ key: (file_info), (data_array, dtype='float')}

The value of the item in the dictionary is split so that the information
and the actually data arrays are kept separate.  Notice my use of
dtype...it is also possible to build your own numpy data type that gives
you a bit more flexibility for storing your data.  This is very useful if
your data is not all that standardized, or if you want to quickly look up
data by reference.  For example, if you have a column in your file called
"counts" and you want later to access this, having a custom datatype will
let you do this with ease.  Anyway, you can read into that later.

This storage type is also highly useful if you need to make new data
structures later.  For example, if you want to plot all of your data in a
multiplot, you can design a method to take this object and return the
formatted multi-array data, as well as any axis arrays that can be
extracted from this data.  Generally, if you can this object built, than
any other representation of the data that you need can be taken from this.
This approach is useful to me, but may not be ideal if your dataset is so
large that you cannot afford to have several data structures that are
holding it simultanesouly in your code.

On Mon, Mar 12, 2012 at 6:23 PM, Abhishek Pratap <apratap at lbl.gov> wrote:

> Super awesome. I love how the python community in general keeps the
> recordings available for free.
>
> @Adam : I do have some problems that I can hit numpy with, mainly
> bigData based. So in summary I have millions/billions of rows of
> biological data on which I want to run some computation but at the
> same time have a capability to do quick lookup. I am not sure if numpy
> will be applicable for quick lookups  by a string based key right ??
>
> -Abhi
>
> On Mon, Mar 12, 2012 at 3:18 PM, Adam Hughes <hugadams at gwmail.gwu.edu>
> wrote:
> > Abhi,
> >
> > One thing I would suggest is to tackle numpy with a particular focus.
> Once
> > you've gotten the basics down through tutorials and videos, do you have a
> > research project in mind to use with numpy?
> >
> >
> > On Mon, Mar 12, 2012 at 6:08 PM, Skipper Seabold <jsseabold at gmail.com>
> > wrote:
> >>
> >> On Mon, Mar 12, 2012 at 6:04 PM, Abhishek Pratap <apratap at lbl.gov>
> wrote:
> >> >
> >> > Hey Guys
> >> >
> >> > Few days with folks at my first pycon has made me wonder how much of
> >> > cool things I was missing ..
> >> >
> >> > I am looking to do some quick catch up on numpy and wondering if there
> >> > are any set of videos that I can refer to. I learn quicker seeing
> >> > videos  and would appreciate if you guys can point me to anything
> >> > available it will be of great help.
> >> >
> >>
> >> You'll find a lot of videos here. The tutorials in particular may
> >> interest you from past conferences.
> >>
> >> http://conference.scipy.org/index.html
> >>
> >> Oddly though it doesn't look like there's a straight link to the 2011
> >> conference there.
> >>
> >> http://conference.scipy.org/scipy2011/
> >>
> >> Skipper
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > NumPy-Discussion at scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20120312/1a030182/attachment.html>