[Python-Dev] Problem between deallocation of modules and func_globals

[Brett Cannon]

On 1/19/07, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2007-01-18 20:53, Brett Cannon wrote:
> > I have discovered an issue relating to func_globals for functions and
> > the deallocation of the module it is contained within.  Let's say you
> > store a reference to the function encodings.search_function from the
> > 'encodings' module (this came up in C code, but I don't see why it
> > couldn't happen in Python code).  Then you delete the one reference to
> > the module that is stored in sys.modules, leading to its deallocation.
> >  That triggers the setting of None to every value in
> > encodings.__dict__.
> >
> > Oops, now the global namespace for that module has everything valued
> > at None.  The dict doesn't get deallocated since a reference is held
> > by encodings.search_function.func_globals and there is still a
> > reference to that (technically held in the interpreter's
> > codec_search_path field).  So the function can still execute, but
> > throws exceptions like AttributeError because a module variable that
> > once held a dict now has None and thus doesn't have the 'get' method.
>
> That's a typical error situation you get in __del__ methods at
> the time the interpreter is shut down.
>

Yeah, but in this case this is at the end of Py_Initialize() for the
stuff I am doing to the interpreter.  =)

> The main reason for setting everything to None first is to
> break circular references and make sure that at least some
> of the object destructors can run.
>

I know the reason, it just happens to occur at a bad time for me.

> > My question is whether this is at all worth trying to rectify.  Since
> > Google didn't turn anything up I am going to guess this is not exactly
> > a common thing.  =)  That would lead me to believe some (probably
> > most) of you will say, "just leave it alone and work around it".
>
> If you can come up with a better way, sure :-)
>
> > The other option I can think of is to store a reference to the module
> > instead of just to its __dict__ in the function.  The problem with
> > that is we end up with a circular dependency of the functions in
> > modules having a reference to the module but then the module having a
> > reference to the functions.  I tried not having the values in the
> > module's __dict__ set to None if the reference count was above 1 and
> > that solved this issue, but that leads to dangling references on
> > anything in that dict that does not have a reference stored away
> > somewhere else like encodings.search_function.
> >
> > Anybody have any ideas on how to deal with this short of rewriting
> > some codecs stuff so that they don't depend on global state in the
> > module or just telling me to just live with it?
>
> I'm not exactly sure which global state you are referring to. The
> aliase map, the cache used by the search function ?
>

encodings._cache .

> Note that the search function registry is a global managed
> in the thread state (it's not stored in any module).
>

Right, but that is not the issue.  If you have deleted the reference
to the encodings module from sys.modules it then sets encodings._cache
to None.  After the deletion, if you try to encode/decode a unicode
string you can an AttributeError about how encodings._cache does not
have a 'get' method since it is now None instead of a dict.  The
function is fine and still runs, it's just that the global state it
depends on is no longer the way it assume it should be.

-Brett

> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, Jan 19 2007)
> >>> Python/Zope Consulting and Support ...        http://www.egenix.com/
> >>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
>
> ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
>