speeding up dictionary creation

[idfx]

In article <399CC5FB.6F4637BD at nowonder.de>,
  Peter Schneider-Kamp <nowonder at nowonder.de> wrote:
> [Bob van der Poel in his original post]
> >
> > it appears to me that when reading/parsing the initial datafile the
> > majority of the time is taken in creating new entries, not in
parsing
> > the file. Is there a way to preallocate entries or otherwise speed
this
> > up. On a short set to data there doesn't seem to be a problem, but
as
> > the list increases in size the load time goes up in what appears to
be
> > N^2.
>

I had a problem a while ago kinda like this.  My solution was also
cPickle.  I also tweaked some extra speed sometimes by using the
dictionary.keys() function, which returns a list, which you can play
with a lot faster.  This also makes incremental updates easier, cuz you
can just search the keyslist for existing data and only have to
actually read and parse and process the part that you have to change...
I got maybe a full 1/3 better speed sometimes, than using the straight
cPickled dictionary, but I dunno if this is at all applicable to your
situation, because my working environment was odd.

('odd' == all elements were strings, and it had to run in a 486 20mHz
machine with a grand total of 2 megs of RAM.  YMMV a lot. )        ;D



--
yrs,
in_sanity,

idfx


Sent via Deja.com http://www.deja.com/
Before you buy.