[Numpy-discussion] Database with Nulls to Numpy Structure

[Mark Janikas]

Thanks for the input!  I wonder if I can resize my own record array?  I.e. one call to truncate... Ill give it a go.  But the resize works great as it doesn't make a copy:

In [12]: a = NUM.arange(10)

In [13]: id(a)
Out[13]: 190182896

In [14]: a.resize(5,)

In [15]: a
Out[15]: array([0, 1, 2, 3, 4])

In [16]: id(a)
Out[16]: 190182896

Whereas the slice seems to make a copy/reassign:

In [18]: a = a[0:2]

In [19]: id(a)
Out[19]: 189981184


Pretty Nice.  Pre-allocate the full space and count number of good records... then resize.  Doesn't seem that much faster than using the lists then creating arrays, but memory should be better.  Thanks again, and anything further would be appreciated.

MJ


-----Original Message-----
From: numpy-discussion-bounces at scipy.org [mailto:numpy-discussion-bounces at scipy.org] On Behalf Of Christopher Barker
Sent: Friday, October 02, 2009 12:34 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Database with Nulls to Numpy Structure

Mark Janikas wrote:
> So, do I use lists and 
> append then create the arrays... Or do I fill up the pre-allocated "empty" 
> arrays and slice off the ends?  Thoughts?  Thanks much...

Either will work. I think the decision would be based on how many Null 
records you expect -- if it's a small fraction then go ahead and 
pre-allocate the array, if it's a large fraction, then you might want to 
go with a list.

Note: you may be able to use arr.resize() to chop it off at the end.

The list method has the downside of using more memory, and being a bit 
slower, which may be mitigated if there are lots of null records.

See an upcoming email of mine for another option...

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion at scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion