[Python-ideas] Proposal how to remove all occurrences of a value from a Python list

[Andrew Barnert]

On Oct 3, 2015, at 14:54, Terry Reedy <tjreedy at udel.edu> wrote:
> 
>> On 10/3/2015 11:05 AM, Eduard Bondarenko wrote:
>> Hello everyone,
>> 
>> the main idea of this proposal is to create convenient, maybe more
>> Pythonic way to remove all occurrences of a value from a Python list.
> 
> This is a special case of in-place filtering.
> 
>> Suppose we have list 'arr':
>> 
>>    arr = [1, 2, 3, 1]
>> 
>> 
>> and we want to remove all 1 from this list.
>> 
>> The most Pythonic way to do it is:
>> 
>> 
>>    arr[:] = (x for x in arr if x != 1)
> 
> arr[:] = filter(lambda x: x != 1, x)

I don't think this is more Pythonic. You've wrapped a perfectly good expression in an unnecessary function just so you can unnecessarily use a higher-order function. (Of course, by the same token, if you already have a perfectly good function lying around, using filter makes more sense than wrapping it up in an unnecessary call expression just so you can avoid a higher-order function. And there are cases where is unclear which is more appropriate--e.g., if what you have lying around is type or instance with a bound or unbound method, or something you can call partial on, is that really better than wrapping it in an expression? But this isn't either of those cases.) I'd have no problem with a novice who didn't yet know comprehensions writing this, but I wouldn't want to teach it as a better alternative to comprehensions in cases where it's not actually better.

>> Looks good, at least for experienced developer, but for Python's
>> newcomers solution will look like this:
>> 
>>     >>> while True:
>>    ...     try:
>>    ...         a.remove(1)
>>    ...     except:
>>    ...         break
> 
> Python programmers really must learn either comprehensions or map and filter.  They must also learn that repeatedly scanning a sequence should be avoided when possible.

Agreed.

In my experience, most novices who ask about code like this, once you explain to them that remove() has to keep repeatedly scanning from the start, immediately see why that's bad and ask how you can avoid doing that. That's a perfect opportunity to teach them about comprehensions while they're looking for exactly what comprehensions can do.

There are some exceptions, but those aren't novices, they're intermediate-experienced C devs who insist that the following is the "simplest" code and must be the fastest or Python is broken (even though it's not necessarily fastest even in C, largely because memmove is so much faster than element by element move):

    idx = skip = 0
    length = len(arr)
    while idx+skip < length:
        if arr[idx+skip] == element_to_remove:
            skip += 1
        else:
            arr[idx] = arr[idx+skip]
            idx += 1
    del arr[idx+1:]

>> Currently I am reading "Effective Python" book and I have encounter good
>> idea that it's also important to have readable code for new or
>> another-language developers. And to my mind current Pythonic 'remove all
>> occurrences' is not readable code and does not give insight (at least at
>> first glance) into what happens in the code.

It's readable to anyone who understands comprehensions and slice assignment. And both of those are such fundamental concepts that, if you don't understand either of them yet at all, your intuitions aren't very good yet.

But notice that there's nothing stopping you from wrapping this up in a function, or adding whitespace or comments to help yourself work through it:

    def remove_all(arr, value):
        """Remove all instances of value from arr"""
        arr[:] = (x for x in arr # keep every element
                  if x != value) # that doesn't equal value

And now, everywhere you use it looks like this:

    remove_all(arr, 1)

And it's hard to imagine anything more readable.

And, even if remove_all isn't the kind of function an experienced developer would write, learning how to factor out the tricky bits into documentable and testable functions is one of the most useful skills for any developer in any language.