[Python-Dev] PEP 0424: A method for exposing a length hint

[Alexandre Zani]

On Sun, Jul 15, 2012 at 8:08 AM, Mark Shannon <mark at hotpy.org> wrote:
> Brett Cannon wrote:
>
>>
>>
>> On Sun, Jul 15, 2012 at 10:39 AM, Mark Shannon <mark at hotpy.org
>> <mailto:mark at hotpy.org>> wrote:
>>
>>     Nick Coghlan wrote:
>>
>>         Right, I agree on the value in being able to return something to
>>         say "this cannot be converted to a concrete container".
>>
>>         I still haven't seen a use case where the appropriate response
>>         to "I don't know" differs from the appropriate response to a
>>         hint of zero - that is, you don't preallocate, you just start
>>         iterating.
>>
>>
>>     There seem to be 5 possible classes values of __length_hint__ that an
>>     iterator object can provide:
>>
>>     1. Don't implement it at all.
>>
>>     2. Implement __length_hint__() but don't want to return any value.
>>        Either raise an exception (TypeError) -- As suggested in the PEP.
>>        or return NotImplemented -- my preferred option.
>>
>>     3. Return a "don't know" value:
>>        Returning 0 would be fine for this, but the VM might want to
>> respond
>>        differently to "don't know" and 0.
>>             __length_hint__() == 0             container should be
>>     minimum size.
>>             __length_hint__() == "unknown"     container starts at
>>     default size.
>>
>>
>>     4. Infinite iterator:
>>        Could return float('inf'), but given this is a "hint" then
>>        returning sys.maxsize or sys.maxsize + 1 might be OK.
>>        Alternatively raise an OverflowError
>>
>>
>> I am really having a hard time differentiating infinity with "I don't
>> know" since they are both accurate from the point of view of __length_hint__
>> and its typical purpose of allocation. You have no clue how many values will
>> be grabbed from an infinite iterator, so it's the same as just not knowing
>> upfront how long the iterator will be, infinite or not, and thus not worth
>> distinguishing.
>>
>>
>>     5. A meaningful length. No problem :)
>>
>>     Also, what are the allowable return types?
>>
>>     1. int only
>>     2. Any number (ie any type with a __int__() method)?
>>     3. Or any integer-like object (ie a type with a __index__() method)?
>>
>>     My suggestion:
>>
>>     a) Don't want to return any value or "don't know": return
>> NotImplemented
>>     b) For infinite iterators: raise an OverflowError
>>     c) All other cases: return an int or a type with a __index__() method.
>>
>>
>> I'm fine with (a), drop (b), and for (c) use what we allow for __len__()
>> since, as Nick's operator.length_hint pseudo-code suggests, people will call
>> this as a fallback if __len__ isn't defined.
>
>
> So how does an iterator express infinite length?
>
> What should happen if I am silly enough to do this:
>>>> list(itertools.count())
>
> This will fail; it should fail quickly.
>
>
> Cheers,
> Mark.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/alexandre.zani%40gmail.com

The PEP so far says: "It may raise a ``TypeError`` if a specific
instance cannot have
its length estimated." In many ways, "I don't know" is the same as
this "specific instance cannot have its length estimated". Why not
just raise a TypeError?

Also, regarding the code Nick posted above, I'm a little concerned
about calling len as the first thing to try. That means that if I
implement both __len__ and __len_hint__ (perhaps because __len__ is
very expensive) __len_hint__ will never be used. It's relatively easy
to say:

try:
  hint = len_hint(l)
except TypeError:
  hint = len(l)