iterating through humongously large key lists!!

[Tim Peters]

[some fragment of code from an older msg]
>>     [snip]
>> newdic = {}
>> while len(oldict):
>>     key, value = oldict.popitem()
>>     [snip]

[Tim reveals <wink> that the memory consumed by oldict doesn't
 decrease in the loop]

[Alex Martelli]
> *Ooops*!  Thanks for clarifying this -- I had failed to notice this
> little "accident of the implementation":-(.

Surely not a failure on anyone's part -- there's no way to know unless you're
familiar with the dict implementation code, and even then it's not obvious.

> OK, this is not a viable way to iterate on a 'humongously large'
> dictionary, then.

In the original example, right, it's not, but there's not enough code left in
the snipped version above to show why:  the original example rebuilt the
oldict into newdic, so ended up with two dicts each consuming as much memory
as the original.

I want to leave people clear on that .popitem()-in-a-loop *is* a viable way
to iterate over a humongously large dict, provided they *don't* mind
destroying the dict in the process.  That's actually the more common case in
my experience, and then iterating via .popitem() works with fixed and small
additional memory.  .popitem() is also ideal for iterating over a dict that
mutates during "the loop"; that's common, e.g., in code using dicts to
represent sets.

not-all-things-to-all-people-but-the-best-thing-for-some-ly y'rs  - tim