For loop searching takes too long!

Jonathan Gardner jgardner at
Mon Feb 1 23:04:01 CET 2010

On Jan 29, 7:07 pm, Steven D'Aprano <st... at REMOVE-THIS-> wrote:
> On Fri, 29 Jan 2010 14:49:06 -0800, Jonathan Gardner wrote:
> > On Jan 28, 3:52 pm, elsa <kerensael... at> wrote:
> >> I've got a problem with my program, in that the code just takes too
> >> long to run. Here's what I'm doing. If anyone has any tips, they'd be
> >> much appreciated!
> > First of all, don't play with large lists. Large lists have a tendency
> > to grow larger over time, until they get absurdly large.
> > If you're dealing with a long list, work with it like you'd work with a
> > file. If you need random access, stick it in some form of database, such
> > as BDB or PostgreSQL. If you need an index, stick it in some form of DB.
> > Eventually, large lists end up like that anyway. Don't fool yourself
> > early on---prepare for the worst and solve the problems of tomorrow
> > today.
> Talk about premature optimization. The OP probably has a few hundreds of
> thousands of items, millions at most, which is trivial on a modern PC.
> Your way of thinking is what has led to Firefox using a database to
> manage a few thousand bookmarks and cookies, which in turn leads to
> consistent data corruption problems. (I've been keeping note, and so far
> my installation of Firefox 3 has had database corruption at startup one
> time out of five.)

I think there's a difference between writing your code so that you
have the option of using a database, or a flat file, or any other type
of serial data source, and actually using a database (or whatever.)

I prefer to write my code so that I don't have to re-write it 6 months

Having worked with datasets that are truly enormous, whenever I see a
list that is unbounded (as this appears to be), I freak out because
unbounded can be a really, really big number. Might as well write
software that works for the truly enormous case and the really large
case and be done with it once and for all.

More information about the Python-list mailing list