[Python-Dev] Problem between deallocation of modules and func_globals
Brett Cannon
brett at python.org
Fri Jan 19 22:33:35 CET 2007
On 1/19/07, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2007-01-18 20:53, Brett Cannon wrote:
> > I have discovered an issue relating to func_globals for functions and
> > the deallocation of the module it is contained within. Let's say you
> > store a reference to the function encodings.search_function from the
> > 'encodings' module (this came up in C code, but I don't see why it
> > couldn't happen in Python code). Then you delete the one reference to
> > the module that is stored in sys.modules, leading to its deallocation.
> > That triggers the setting of None to every value in
> > encodings.__dict__.
> >
> > Oops, now the global namespace for that module has everything valued
> > at None. The dict doesn't get deallocated since a reference is held
> > by encodings.search_function.func_globals and there is still a
> > reference to that (technically held in the interpreter's
> > codec_search_path field). So the function can still execute, but
> > throws exceptions like AttributeError because a module variable that
> > once held a dict now has None and thus doesn't have the 'get' method.
>
> That's a typical error situation you get in __del__ methods at
> the time the interpreter is shut down.
>
Yeah, but in this case this is at the end of Py_Initialize() for the
stuff I am doing to the interpreter. =)
> The main reason for setting everything to None first is to
> break circular references and make sure that at least some
> of the object destructors can run.
>
I know the reason, it just happens to occur at a bad time for me.
> > My question is whether this is at all worth trying to rectify. Since
> > Google didn't turn anything up I am going to guess this is not exactly
> > a common thing. =) That would lead me to believe some (probably
> > most) of you will say, "just leave it alone and work around it".
>
> If you can come up with a better way, sure :-)
>
> > The other option I can think of is to store a reference to the module
> > instead of just to its __dict__ in the function. The problem with
> > that is we end up with a circular dependency of the functions in
> > modules having a reference to the module but then the module having a
> > reference to the functions. I tried not having the values in the
> > module's __dict__ set to None if the reference count was above 1 and
> > that solved this issue, but that leads to dangling references on
> > anything in that dict that does not have a reference stored away
> > somewhere else like encodings.search_function.
> >
> > Anybody have any ideas on how to deal with this short of rewriting
> > some codecs stuff so that they don't depend on global state in the
> > module or just telling me to just live with it?
>
> I'm not exactly sure which global state you are referring to. The
> aliase map, the cache used by the search function ?
>
encodings._cache .
> Note that the search function registry is a global managed
> in the thread state (it's not stored in any module).
>
Right, but that is not the issue. If you have deleted the reference
to the encodings module from sys.modules it then sets encodings._cache
to None. After the deletion, if you try to encode/decode a unicode
string you can an AttributeError about how encodings._cache does not
have a 'get' method since it is now None instead of a dict. The
function is fine and still runs, it's just that the global state it
depends on is no longer the way it assume it should be.
-Brett
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source (#1, Jan 19 2007)
> >>> Python/Zope Consulting and Support ... http://www.egenix.com/
> >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
> ________________________________________________________________________
>
> ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
>
More information about the Python-Dev
mailing list