For loop searching takes too long!

[Steven D'Aprano]

On Mon, 01 Feb 2010 14:04:01 -0800, Jonathan Gardner wrote:

> Having worked with datasets that are truly enormous, whenever I see a
> list that is unbounded (as this appears to be), I freak out because
> unbounded can be a really, really big number. Might as well write
> software that works for the truly enormous case and the really large
> case and be done with it once and for all.


Then write it to work on iterators, if at all possible, and you won't 
care how large the data stream is. Even databases have limits!

Of course you should write your code to deal with your expected data, but 
it is possible to commit overkill as well as underkill. There are trade-
offs between development effort, runtime speed, memory usage, and size of 
data you can deal with, and maximizing the final one is not always the 
best strategy.

Look at it this way... for most applications, you probably don't use 
arbitrary precision floats even though floating point numbers can have an 
unbounded number of decimal places. But in practice, you almost never 
need ten decimal places to pi, let alone an infinite number. If the 
creators of floating point libraries wrote software "that works for the 
truly enormous case", the overhead would be far too high for nearly all 
applications.

Choose your algorithm according to your expected need. If you under-
estimate your need, you will write slow code, and if you over-estimate 
it, you'll write slow code too. Getting that balance right is why they 
pay software architects the big bucks *wink*



-- 
Steven