Populating a dictionary, fast [SOLVED SOLVED]
Michael Bacarella
mbac at gpshopper.com
Thu Nov 15 15:51:25 EST 2007
> On Nov 15, 2:11 pm, Istvan Albert <istvan.alb... at gmail.com> wrote:
> > There is nothing wrong with neither creating nor deleting
> > dictionaries.
>
> I suspect what happened is this: on 64 bit
> machines the data structures for creating dictionaries
> are larger (because pointers take twice as much space),
> so you run into memory contention issues sooner than
> on 32 bit machines, for similar memory sizes.
> If there is something deeper going
> on please correct me, I would very much like to know.
Since some people missed the EUREKA!, here's the executive summary:
Python2.3: about 45 minutes
Python2.4: about 45 minutes
Python2.5: about _30 seconds_
The cut/paste of the EUREKA MOMENT from earlier in this thread:
> > You can download the list of keys from here, it's 43M gzipped:
> > http://www.sendspace.com/file/9530i7
> >
> > and see it take about 45 minutes with this:
> >
> > $ cat cache-keys.py
> > #!/usr/bin/python
> > v = {}
> > for line in open('keys.txt'):
> > v[long(line.strip())] = True
> >
> >
> It takes about 20 seconds for me. It's possible it's related to
> int/long
> unification - try using Python 2.5. If you can't switch to 2.5,
try
> using string keys instead of longs.
Yes, this was it. It ran *very* fast on Python v2.5.
Terribly on v2.4, v2.3.
(I thought I had already evaluated v2.5 but I see now that the
server
With 2.5 on it invokes 2.3 for 'python'.)
Thanks!
More information about the Python-list
mailing list