unique-ifying a list

[Elias Fotinis (eliasf)]

"kj" wrote:
> I suppose that I could write something like
>
> def uniquify(items):
>    seen = set()
>    ret = []
>    for i in items:
>        if not i in seen:
>            ret.append(i)
>            seen.add(i)
>    return ret
>
> But this seems to me like such a commonly needed operation that I
> find it hard to believe one would need to resort to such self-rolled
> solutions.  Isn't there some more standard (and hopefully more
> efficient, as in "C-coded"/built-in) approach?

The most "standard" way is a recipe from the itertools docs (I'd give a 
link, but python.org is down at the moment):

    def unique_everseen(iterable, key=None):
        "List unique elements, preserving order. Remember all elements ever 
seen."
        # unique_everseen('AAAABBBCCDAABBB') --> A B C D
        # unique_everseen('ABBCcAD', str.lower) --> A B C D
        seen = set()
        seen_add = seen.add
        if key is None:
            for element in iterable:
                if element not in seen:
                    seen_add(element)
                    yield element
        else:
            for element in iterable:
                k = key(element)
                if k not in seen:
                    seen_add(k)
                    yield element

All the recipes mentioned there are pretty handy, so I've made a module 
(iterutil.py) out of them.