mysql -> record array

[Travis Oliphant]

John Hunter wrote:

>>>>>>"John" == John Hunter <jdhunter at ace.bsd.uchicago.edu> writes:
>>>>>>            
>>>>>>
>
>  
>
>>>>>>"Erin" == Erin Sheldon <erin.sheldon at gmail.com> writes:
>>>>>>            
>>>>>>
>    Erin> The question I have been asking myself is "what is the
>    Erin> advantage of such an approach?".  It would be faster, but by
>
>    John> In the use case that prompted this message, the pull from
>    John> mysql took almost 3 seconds, and the conversion from lists
>    John> to numpy arrays took more that 4 seconds.  We have a list of
>    John> about 500000 2 tuples of floats.
>
>    John> Digging in a little bit, we found that numpy is about 3x
>    John> slower than Numeric here
>
>    John>   peds-pc311:~> python test.py with dtype: 4.25 elapsed
>    John> seconds w/o dtype 5.79 elapsed seconds Numeric 1.58 elapsed
>    John> seconds 24.0b2 1.0.1.dev3432
>
>    John> Hmm... So maybe the question is -- is there some low hanging
>    John> fruit here to get numpy speeds up?
>
>And for reference, numarray is 5 times faster than Numeric here and 15
>times faster than numpy
>
>  peds-pc311:~> python test.py
>  with dtype: 4.20 elapsed seconds
>  w/o dtype 5.71 elapsed seconds
>  Numeric  1.60 elapsed seconds
>  numarray  0.30 elapsed seconds
>  24.0b2
>  1.0.1.dev3432
>  1.5.1
>
>import numarray
>tnow = time.time()
>y = numarray.array(x, numarray.Float)
>tdone = time.time()
>  
>
This sounds like it could be definitely be sped up, then.      
Assign_Array is the relevant code (it then calls PySequence_SetItem) so 
that basically

for k in a.shape[0]:
      a[k] = x[k]

is what is being done.  Thus, it might be the indexing code that is 
causing this to be a slower operation.  We should look at what numarray 
is doing --- it has provided important speed-ups in the past.

I still don't have time to look at this, but please file a ticket as we 
should fix this one.  Reference the faster numarray implementation.

-Travis



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV