On Sun, May 30, 2021 at 9:10 AM Julien Palard email@example.com wrote:
is the fact some things (like generators) give iterators instead of iterables as a hint they're not "rewindable" was initially thought of and part of the design, or it emerged later.
Hm... I don't think that was a big part of the original design. The true difference between iterable and iterator is that the iterator stores the state needed to iterate over a given iterable with a for-loop. So if you have an array, and you have two loops over them (e.g. nested, like this:
for x in a: for y in a: print(x, y, x+y)
) then you need separate iterator objects so that advancing the inner iterator doesn't affect the outer iterator. This is why you can't store the iteration state in the iterable (the array) but must use a separate object.
Iterators themselves cannot rewind -- at least, there's no standard API for it, and although nothing stops you from adding such an API to a *specific* iterator type, it's not a common pattern.
Returning "self" as the iterator was originally only intended to paper over the case where you want to write
it = iter(a) <maybe call next(it) a few times> for x in it: ...
-- basically we wanted 'for x in iter(a)' and 'for x in a' to have the same meaning.
IIRC iterables returning "self" as the iterator in other cases first came up for files, where we had long been struggling to find the best API to get all the lines of the file while still benefiting from buffering (calling f.readline() in a loop was too slow).
The first version of this API was f.readlines(), which returned a list of strings. But we realized this could potentially use up too much memory, so we added an optional "hint" argument so you could say f.readlines(100000) and get a number of lines approximately corresponding to 100000 bytes. This required people to write fairly tedious double loops to loop over all lines efficiently, e.g.
while 1: lines = f.readlines(100000) if not lines: break for line in lines: <do the thing per line>
Maybe there was an intermediate step (I vaguely recall a special dunder?), but eventually we realized that the best way to write this was just
for line in f: <do the thing per line>
(the iterator can buffer internally) and we accepted that you can only iterate once over a file -- we just told people "if you double-iterate over a file it doesn't work right".