Huge Dicts and perfomance

[Brian Kelley]

I don't know if this affects you but:

There is a relatively easy way to increase dictionary performance.

I use dictionaries to store graph formats with edges and vertices.  I 
have found that using an instance as the dictionary lookup is quite 
slow.  Instead, I use integer handles as the lookup.

Attached is a piece of code that hashes on instances and also hashes on 
a generated integer handle.  Each node object has an attribute 
node.handle which is unique for every node.

Note that using __hash__ to return the handle doesn't speed things up! 
Python has to do a *lot* of machinations to see if two instances are the 
same thing.  For an order of magnitude speed up for using objects as 
dictionary keys, use handles instead.  You just have to make sure that 
the handles are unique :)

In the test code included, I have noticed that using integer handles 
instead of instances speeds lookup by a factor of 20.

See the attached code for a better idea of what I mean.

I hope this helps.

Brian
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test_handles.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20011220/9634208c/attachment.ksh>