[Python-Dev] PEP 0424: A method for exposing a length hint

[Steven D'Aprano]

Mark Shannon wrote:

> So how does an iterator express infinite length?

The suggestion was it should return -1.


> What should happen if I am silly enough to do this:
>  >>> list(itertools.count())
> 
> This will fail; it should fail quickly.

That depends on your OS. I've just tested it now on Linux Mint, and the Python 
process was terminated within seconds.

I've also inadvertently done it on a Fedora system, which became completely 
unresponsive to user-input (including ctrl-alt-delete) within a few minutes. I 
let it run overnight (16 hours) before literally pulling the plug.

(I expect the difference in behaviour is due to the default ulimit under 
Debian/Mint and RedHat/Fedora systems.)

Ignoring OS-specific features, the promise[1] of the language is that list 
will try to allocate enough space for every item yielded by the iterator, or 
fail with a MemoryError. No promise is made as to how long that will take: it 
could take hours, or days, depending on how badly memory allocation 
performance drops when faced with unreasonably large requests. You can't 
expect it to fail either quickly or with an exception.

With a length hint, we could strengthen that promise:

"if __length_hint__ returns a negative number, list, tuple and set will fail 
immediately with MemoryError"

which I think is a good safety feature for some things which cannot possibly 
succeed, but risk DOSing your system. Does it prevent every possible failure 
mode? No, of course not. But just because you can't prevent *every* problem 
doesn't mean you should prevent the ones which you can.


[1] I think. I'm sure I read this somewhere in the docs, but I can't find it now.


-- 
Steven