[Python-Dev] PEP 0424: A method for exposing a length hint
steve at pearwood.info
Sun Jul 15 17:55:09 CEST 2012
Mark Shannon wrote:
> So how does an iterator express infinite length?
The suggestion was it should return -1.
> What should happen if I am silly enough to do this:
> >>> list(itertools.count())
> This will fail; it should fail quickly.
That depends on your OS. I've just tested it now on Linux Mint, and the Python
process was terminated within seconds.
I've also inadvertently done it on a Fedora system, which became completely
unresponsive to user-input (including ctrl-alt-delete) within a few minutes. I
let it run overnight (16 hours) before literally pulling the plug.
(I expect the difference in behaviour is due to the default ulimit under
Debian/Mint and RedHat/Fedora systems.)
Ignoring OS-specific features, the promise of the language is that list
will try to allocate enough space for every item yielded by the iterator, or
fail with a MemoryError. No promise is made as to how long that will take: it
could take hours, or days, depending on how badly memory allocation
performance drops when faced with unreasonably large requests. You can't
expect it to fail either quickly or with an exception.
With a length hint, we could strengthen that promise:
"if __length_hint__ returns a negative number, list, tuple and set will fail
immediately with MemoryError"
which I think is a good safety feature for some things which cannot possibly
succeed, but risk DOSing your system. Does it prevent every possible failure
mode? No, of course not. But just because you can't prevent *every* problem
doesn't mean you should prevent the ones which you can.
 I think. I'm sure I read this somewhere in the docs, but I can't find it now.
More information about the Python-Dev