[Numpy-discussion] How to convert a list into a structured array?

[Wei Su]

Hi, Francesc:
 
Thanks a lot for offering me help. My code is really simple as of now.
 
**********************************************************************************
from pyodbc import *
from rpy import *

cnxn = connect('DRIVER={SQL Server};SERVER=srdata01\\sql2k5;DATABASE=Qai;UID=;PWD=')
cursor = cnxn.cursor()

cursor.execute("select IsrCode, MstrName from qai..qaiLinkBase")
data = cursor.fetchall()

cursor.close()
***************************************************

The result, data, I got from the above code tends to be a giant list, which is very hard to handle. My goal is to to turn it into a record array so that i can access the field directly by name or by index. My data is typically numerical, character and datetime variables. no other complications.
 
>From the above code, you can also see that I used R for some time. But I have to switch to something else because I sometimes cannot even download all my data via R due to its memory limit under windows. I thought NumPy might be the solution. But I am not sure. Anybody can let me know whether Python has a memory limit? or can I use virtual memory by calling some Python module?
 
Thanks in advance.
 
Wei  Su
 
 

--- On Tue, 5/5/09, Francesc Alted <faltet at pytables.org> wrote:


From: Francesc Alted <faltet at pytables.org>
Subject: Re: [Numpy-discussion] How to convert a list into a structured array?
To: "Discussion of Numerical Python" <numpy-discussion at scipy.org>
Date: Tuesday, May 5, 2009, 7:10 AM


Welcome Wei!

A Monday 04 May 2009, Wei Su escrigué:
> Hi,All:
>  
> My first post! I am very excited to find out structured array (record
> array) in Python. Since I do data manipulation every day, this is
> truly great. However, I typically download data using pyodbc, the
> default output is a big list. So I am wondering how to convert that
> big list into a structured array? using array() will turn it into a
> text array, afaik. it is even better if anybody can show me some
> tricks to download the data directly as a structured array. 
> Thanks a lot for the help.

Please, could you provide an example of the list that you are getting 
from your database?  With that we can probably figure out your needs 
much better.

> BTW: I am also interested in Python's ability to handle large data.
> Any hints or suggestion is welcome.

This is also a bit generic question.  What kind of data you have to deal 
with?  What sort of operations do you want to perform over it?  Do you 
need a lot of speed or flexibility is more important?  Some example?

Cheers,

-- 
Francesc Alted

"One would expect people to feel threatened by the 'giant
brains or machines that think'.  In fact, the frightening
computer becomes less frightening if it is used only to
simulate a familiar noncomputer."

-- Edsger W. Dykstra
   "On the cruelty of really teaching computer science"
_______________________________________________
Numpy-discussion mailing list
Numpy-discussion at scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090505/0cb21d7c/attachment.html>