<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=ISO-8859-1" />
<title>Most efficient way to build very large dictionaries</title>
</head>
<body dir="ltr">
I'm working with some very large dictionaries (about 25M items per dictionary) that get filled based on data parsed from text based log files. I'm using Python dictionaries to track the frequency of specific combinations of events, eg, I build a key and then add various timing info from the current record to that's key's current value. If the key doesn't exist, I create it and initalize it to its first value. Most keys will have anywhere from 2 to 10,000 value increments.<br /><br />Over time, I've noticed that 90%+ of the keys in my dictionaries stay constant across daily runs of my program. Can I take advantage of this knowledge to optimize my dictionary performance by pre-building my dictionary based on a given list of keys whose values are all set to 0? Specifically, is there a way to bulk load a dictionary with keys/initial values that is faster than individually adding most dictionary entries one at a time?<br /><br />Here are 2 high level strategies I've been thinking about for improving the performance of my dictionaries.<br /><br />1. Create a list of keys from a current dictionary, pickle this list of keys and then on my next program run, load the pickled list of keys and somehow convert this list to a dictionary with all key values set to 0.<br /><br />2. After I've performed my analysis on a current dictionary, iterate through this dictionary, setting all values to 0 (is there a fast way to do this?), and then on my next program run, load this pickled dictionary of keys and zero values.<br /><br />I would appreciate your thoughts on whether there are advantages to working with a pre-built dictionary and if so, what are the best ways to create a pre-loaded dictionary.<br /><br />Thank you,<br />Malcolm<br />
</body>
</html>