[Python-Dev] Nested scopes core dump

[Michael Hudson]

Guido van Rossum <guido@digicool.com> writes:

> > >   MH> The dict's resizing, it turns out.
> > > 
> > > So a hack to make the iteration safe would be to assign and element
> > > and then delete it?
> > 
> > Yes.  This would be gross beyond belief though.  Particularly as the
> > normal case is for freevars to be empty.
> > 
> > >   MH> I note that in PyDict_SetItem, the check to see if the dict
> > >   MH> needs resizing occurs *before* it is known whether the key is
> > >   MH> already in the dict.  But if this is the problem, how come we
> > >   MH> haven't been bitten by this before?
> > > 
> > > It's probably unusual for a dictionary to be in this state when the
> > > compiler decides to update the values.
> > 
> > What I meant was that there are bits and pieces of code in the Python
> > core that blithely pass keys gotten from PyDict_Next into
> > PyDict_SetItem.
> 
> Where?

import.c:PyImport_Cleanup
moduleobject.c:_PyModule_Clear

Hrm, I was sure there were more than that, but there don't seem to be.
Sorry for the alarmism.

> > From what I've just learnt, I'd expect this to
> > occasionally cause glitches of extreme confusing-ness.  Though on
> > investigation, I don't think any of these bits of code are sensitive
> > to getting keys out multiple times (which is what happens in this case
> > - though you must be able to miss keys too).  Might cause the odd leak
> > here and there.
> 
> I'd fix the dict implementation, except that that's tricky.

I'd got that far...

> Checking for a dup key in PyDict_SetItem() before calling dictresize()
> slows things down.  Checking in insertdict() is wrong because
> dictresize() uses that!

Maybe you could do the check for resize *after* the call to
insertdict?  I think that would work, but I wouldn't like to go
messing with such a performance critical bit of code without some
careful thinking.

Cheers,
M.

-- 
  You sound surprised.  We're talking about a government department
  here - they have procedures, not intelligence.
                                            -- Ben Hutchings, cam.misc