On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters <thomas@python.org> wrote:

On Wed, Jun 5, 2013 at 3:20 AM, lukasz.langa <python-checkins@python.org>wrote:

...

+from weakref import WeakKeyDictionary

FYI, this change exposes a bug in the atexit module involving subinterpreters, causing the refleaks reported by Antoine's daily report: interpreter startup now always imports weakref, which imports atexit and registers a classmethod. Unfortunately the atexit module doesn't seem to know subinterpreters from subtitles and so doesn't unregister this classmethod when the subinterpreter is terminated.

This isn't a new bug, but it's exposed by always importing weakref and atexit during interpreter startup. I'm wondering if that's really necessary :)

Is there an issue tracking this? -Brett

-- Thomas Wouters <thomas@python.org>

Hi! I'm an email virus! Think twice before sending your email to help me spread!

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org

On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters <thomas@python.org> wrote:

This isn't a new bug, but it's exposed by always importing weakref and atexit during interpreter startup. I'm wondering if that's really necessary :)

Importing it during startup isn't necessary per se; imports needed only by the generic function implementation can and should be imported late, rather than at the time functools is imported. However, if pkgutil was/is migrated to using this implementation of generics, then it's likely going to end up imported during startup anyway, because at least the -m startup path involves pkgutil. In short, the overall answer right now is, "maybe", and the answer later is "rather likely". ;-)

On Fri, Jun 7, 2013 at 5:16 PM, Łukasz Langa <lukasz@langa.pl> wrote:

On 7 cze 2013, at 22:50, PJ Eby <pje@telecommunity.com> wrote:

...

On Fri, Jun 7, 2013 at 10:27 AM, Thomas Wouters <thomas@python.org> wrote:

...

This isn't a new bug, but it's exposed by always importing weakref and atexit during interpreter startup. I'm wondering if that's really necessary :)

In short, the overall answer right now is, "maybe", and the answer later is "rather likely". ;-)

I would rather say that it's "rather certain".

functools is necessary for setup.py to work while bootstrapping, whereas pkgutil is used in runpy.py which is always imported in Modules/main.c.

So we're left with having to fix atexit to support subinterpreters. I wonder how difficult that will be.

If the problem really has to do with interpreter startup, then there actually is a workaround possible, at the cost of slightly hairier code. If dispatch() looked in the registry *first* and avoided the cache in that case, and lazily created the cache (including the weakref import), then interpreter startup would not trigger an import of weakref in the default case. (Of course, depending on whether site/sitecustomize results in the use of importer subclasses and such, this might not help. It might be necessary to take the even more complex tack of avoiding the use of a cache entirely until an ABC is registered, and walking mro's.) Anyway, it remains to be seen whether these workarounds are easier or more difficult than fixing the atexit problem. ;-) Hmm... actually, there are a couple other ways around this. singledispatch doesn't use finalize(), so it doesn't really need atexit. It doesn't even do much with WeakKeyDictionary, so it could actually just "from _weakref import ref", and inline the relevant operations. Or, WeakKeyDictionary could be pulled out into a separate module, where singledispatch could pull it from without importing finalize. Or, weakref.finalize could be fixed so that the atexit import and register() are deferred until actual use. (Of all of these, that last one actually sounds like the least invasive workaround, with fewest lines of code likely to be changed.)