Best way to handle large lists?

[Duncan Booth]

Chaz Ginger <cginboston at hotmail.com> wrote:

> I have a system that has a few lists that are very large (thousands or
> tens of thousands of entries) and some that are rather small. Many times
> I have to produce the difference between a large list and a small one,
> without destroying the integrity of either list. I was wondering if
> anyone has any recommendations on how to do this and keep performance
> high? Is there a better way than
> 
> [ i for i in bigList if i not in smallList ]

How about:

smallSet = set(smallList)
something = [ i for i in bigList if i not in smallSet ]

Use timeit.py on some representative data to see what difference that 
makes.