[Python-Dev] Showstopper

[Tim Peters]

[Guido, see last point, about dict.keys()/.values()]

[Michael Hudson]
> Crap.

An accurate summary <wink>.

> Two ideas suggest themselves: (1) don't have the resize check in
> PyDict_Next (it could be in PyDict_SetItem instead, though the fact
> that this is safe is delicate to say the least) (2) don't use
> PyDict_Next in dict_traverse.

See preceding crossed-in-the-mail msg.

> OTOH, the GC runs __del__ methods, right?  So what if a __del__ method
> mutates the dictionary that .items() is being called on?  If
> allocating memory can execute arbitrary Python code,

Right.

> I dread to think how many bugs of this form are hiding in Python
> (actually it's only allocating containers that's the worry, but
> still...).

I did too at first, but it appears to be tractable:  allocating containers
"in the middle" of something else is actually rare.  OTOH, listobject.c
eventually grew a faux "immutable list type", after an endless sequence of
hacks failed to stop all cases in which list.sort() could be tricked into
blowing up by writing devious comparison functions that mutated the list in
"just the right way" during the sort.  Today you get an exception if you try
to mutate a list while it's being sorted, and that worked great (I confess
I'm much fonder of it than Guido is, though).

I think we can stop disasters in the dict code, but also expect "odd
behavior" will be possible; e.g., that dict.items() may return a list with
duplicate keys if code is insane enough to mutate the dict during a __del__
method (or in another thread:  once we execute __del__, we're executing
Python code, and the eval loop will let other threads in).

> On the third hand, I can't trigger one deliberately, so maybe
> I'm talking nonsense.

I expect these are very difficult to trigger.

> To fix items/keys/values, you could build up the list of tuples first,
> check you still have the right amount, then fill them in.

Yes, that's one of the things Guido intends to do, although we only talked
about dict_items().  keys() and values() don't allocate any tuples.  They do
allocate a result list at the start, but-- sigh! --you're right that
mp->ma_used may change "during" the initial

    v = PyList_New(mp->ma_used);

> not-sure-this-is-helping-ly y'rs

it-was-depressing-so-it-must-be-helpful<wink>-ly y'rs  - tim