very large dictionary

Steven D'Aprano steven at
Tue Aug 5 11:47:57 CEST 2008

On Tue, 05 Aug 2008 01:20:08 -0700, Simon Strobl wrote:

>> > I thought it would be practical not to create the dictionary from a
>> > text file each time I needed it. I.e. I thought loading the .pyc-file
>> > should be faster. Yet, Python failed to create a .pyc-file
>> Probably a good example of premature optimization.
> Well, as I was using Python, I did not expect to have to care about the
> language's internal affairs that much. I thought I could simply do
> always the same no matter how large my files get. In other words, I
> thought Python was really scalable.

Yeah, it really is a pain when abstractions leak.

>> Out of curiosity, how
>> long does it take to create it from a text file?
> I do not remember this exactly. But I think it was not much more than an
> hour.

Hmmm... longer than I expected. Perhaps not as premature as I thought. 
Have you tried the performance of the pickle and marshal modules?


More information about the Python-list mailing list