Removing dupes from a list

[Cliff Wells]

On Fri, 26 Apr 2002 22:39:46 +0200
Alexander Skwar wrote:

> Hi!
> 
> What's the fastest way to make sure that a list only contains unique
> entries?
> 
> Currently, I get all the elements from the list and store them as keys
> in a dictionary.  I then use dict.keys() as the new value for the list,
> like so:
> 
> def de_dupe(liste):
>         d = {}
>         for element in liste:
>                 d[element] = None
>         return list(d)
> 
> But this will obviously only work, if all the elements of list are
> immutable.

Well, there's probably a faster way, but this is fairly simple:

def de_dup(liste):
    dups = filter(lambda i: liste.count(i) > 1, liste)
    return [i for i in liste if i not in dups]

>>> l = [[1,2,3],[1,2,3],[4,5],[1,2],[1,2,3], 1, 2, 3, 4, 2]
>>> de_dup(l)
[[4, 5], [1, 2], 1, 3, 4]
>>> 

I expect the list.count() will be a performance killer on large lists.

-- 
Cliff Wells, Software Engineer
Logiplex Corporation (www.logiplex.net)
(503) 978-6726 x308  (800) 735-0555 x308