[Baypiggies] sorting a table by column

Chris Clark Chris.Clark at ingres.com
Thu Jun 23 19:31:44 CEST 2011


Tripathi, Bibha wrote:
>
> a huge table like an excel sheet, saved and accumulating more data 
> more rows, may be more tables
> user chooses which column to sort on
>
> what's the best python data structure to use?

It probably depends on what "huge" means. If the "table" data fits in 
memory (either physical or virtual) it probably isn't that big and doing 
operations in Python (e.g. using list comprehension) is probably 
appropriate. If it doesn't fit in memory you can't easily use list 
comprehension and need to look into loops and generators, see 
http://danielrech.net/2011/python-generators-presentation-by-david-beazley/ 
(I think Alex might have done something similar at PyCon a few years ago 
too).

There are some third party libs on PyPi that are worth checking out. I 
keep wanting to find an excuse to kick the tires on 
http://pypi.python.org/pypi/blist/ but I've not had cause to do so yet. 
Search PyPi for "tree" and there are a lot of hits.

You can even use good old Schwartzian transforms (aka 
decorate-sort-undecorate) to handle changes in columns if for some 
reason there isn't a key argument to sort() provided by the structure 
you choose.


David Berthelot wrote:
> Looks like a typical SQL problem.
>   

Agreed, without more information this sounds like a classic "ORDERY BY" 
clause on a SELECT statement. Relational database really excel, lower 
case "e", rather than upper case "E" :-), at this sort of thing..... If 
you check my email address domain name, of course I'm going to say that 
;-) Databases do a lot of the heavy lifting for you.

If you're doing some sort of BI analytics a database is probably your 
best bet (I'm taking a guess you are based on your email address domain 
name). Shameless promotion, take a gander at 
http://www.thevirtualcircle.com/2011/02/vectorwise-theres-a-disturbance-in-the-force/ 
and http://www.ingres.com/products/vectorwise (I don't work on 
Vectorwise but I'm always blown away at how fast it is). If you are 
avoiding a traditional DBMS for performance reasons VW may well surprise 
you..

Chris



More information about the Baypiggies mailing list