[IPython-dev] Nasty, nasty references bug

[Fernando Perez]

Hi all,

On Sat, Apr 4, 2009 at 9:23 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> Hi all,
>
> https://bugs.launchpad.net/ipython/+bug/269966 has proven to be a
> rather unpleasant little beast.
>
> I've attached here a file that explains in great detail the problem
> and illustrates it even without using ipyton at all.  The issue, at
> heart, has to do with how the Python VM tears down module objects when
> they go out of scope.  I suspect the module teardown machinery is
> allowed to be so aggressive (it takes a rather Jack the Ripper
> approach to the objects contained in a module, without any regard for
> their reference count) because in normal use, modules are only deleted
> at interpreter shutdown.  By that point all politeness tends to be
> cast overboard and the interpreter just goes on a rampage on anything
> it finds in memory.

OK, this proved to be rather painful, but I think that I have
something that works.  I was able to add some tests to the suite that
fail under this particular error, and then implemented a solution I'd
thought of on Friday but hadn't had the chance to try:

https://code.launchpad.net/~fdo.perez/ipython/trunk-dev

It's not particularly elegant, but it's better than any alternative I
can think of.  I basically made the module where %run executes be
created only once, but I clear its __dict__ and cache that contents on
each invocation.  Having that object NOT be a true module breaks
inspect (and we'd have to monkeypatch it, which we've done in the past
but has its own share of problems).  But if it's a real module that is
allowed to be deleted, then Python's Jack the Module Ripper kicks in
and leaves us with broken references.  So my idea was to avoid both:
it's a real module, it's never deleted, but its execution namespace is
cleared and cached elsewhere as a normal dict.

It's a bit of a tap dance to make it all work, but I made it all
internal to the object so external users (see the changes to %run)
only see a civilized API.

This was massively problematic, because either form of the bug was a
real problem:

- uncontrolled memory leak on %run
- broken references on successive uses of %run

I do think that now we have a real solution.

Once the changes are reviewed so I can merge them, I'd appreciate
everyone who is willing to run from trunk to let us know if they see
any problems with heavy use of %run.

I think this was the other major blocker we had (given that we also
have an ugly-but-usable workaround for the ipdoctest/twisted issue).
I have some urgent things at work for the next 2 days so I'll be
mostly offline, but I'll do my best to review all pending branches so
we can put an RC out for 0.10.

Sorry this took so long, but these two bugs (the ipdoctest/twisted and
the references one) were truly horrid.

Cheers,

f