[Python-bugs-list] [ python-Bugs-665835 ] filter() treatment of str and tuple inconsistent

[SourceForge.net]

Bugs item #665835, was opened at 2003-01-10 11:36
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=665835&group_id=5470

Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Walter Dörwald (doerwalter)
Assigned to: Raymond Hettinger (rhettinger)
Summary: filter() treatment of str and tuple inconsistent

Initial Comment:
class tuple2(tuple):
·  def __getitem__(self, index):
·  ·  return 2*tuple.__getitem__(self, index)

class str2(str):
·  def __getitem__(self, index):
·  ·  return chr(ord(str.__getitem__(self, index))+1)

print filter(lambda x: x>1, tuple2((1, 2)))
print filter(lambda x: x>"a", str2("ab"))

this prints:
(2,)
bc

i.e. the overwritten __getitem__ is ignored in the
first case, but honored in the second.


----------------------------------------------------------------------

>Comment By: Raymond Hettinger (rhettinger)
Date: 2003-01-25 11:45

Message:
Logged In: YES 
user_id=80475

None of the existing iterators (incl dicts, lists, tuples, and 
files) use __getitem__.  Most likely, user defined iterators 
also access the data structure directly (for flexiblity and 
speed). Also, anything that uses PyTuple_GET_ITEM 
bypasses __getitem__.

If string/unicode iterators are added, they should also go
directly to the underlying data; otherwise, there is no point 
to it.

Also, the proposal to change filtertuple(), doesn't solve
inconsistencies within filterstring() which uses __getitem__ 
when there is a function call, but bypasses it when the 
function parameter is Py_None.

I think the right answer is to change filterstring() to use an 
iterator and to implement string/unicode iterators that 
access the data directly (not using __getitem__).

FYI for Tim:  MvL noticed and fixed the unicode vs string 
difference.  His patch, SF #636005, has not been applied 
yet.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2003-01-25 08:51

Message:
Logged In: YES 
user_id=6380

(But in addition th that, I don't mind having a custom
string iterator -- as long as it calls __getitem__ properly.
Hm, shouldn't the tuple iterator call __getitem__ properly too?)


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-01-25 08:45

Message:
Logged In: YES 
user_id=31435

Just noting that filter() is unique in special-casing the type 
of the input.  It's always been surprising that way, and, 
e.g., filtering a string produces a string, but filtering a 
Unicode string produces a list.

map() and reduce() don't play games like that, and always 
use the iteration protocol to march over their inputs.

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2003-01-25 08:36

Message:
Logged In: YES 
user_id=6380

I don't know which Python sources Raymond has been reading,
but in the sources I've got in front of me, there are
special cases for strings and tuples, and these *don't* use
iter(). It so happens that the tuple special-case calls
PyTuple_GetItem(), which doesn't call your __getitem__,
while the string special-case calls the sq_item slot
function, which (in your case) will be a wrapper that calls
your __getitem__.

A minimal fix would be to only call filtertuple for strict
tuples -- although this changes the output type, but I don't
think one should count on filter() of a tuple subclass
returning a tuple (and it can't be made to return an
instance of the subclass either -- we don't know the
constructor signature).

Similar fixes probably need to be made to map() and maybe
reduce().

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2003-01-24 22:47

Message:
Logged In: YES 
user_id=80475

The problem isn't with filter() which correctly calls iter() in 
both cases.

Tuple object have their own iterator which loops over 
elements directly and has no intervening calls to 
__getitem__().

String objects do not define a custom iterator, so iter() 
wraps itself around consecutive calls to __getitem__().

The resolution is to provide string objects with their own 
iterator. As a side benefit, iteration will run just a tiny bit 
faster.  The same applies to unicode objects.

Guido, do you care about this and want me to fix it or 
would you like to close it as "won't fix".


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=665835&group_id=5470