[Python-Dev] Evil reference cycles caused Exception.__traceback__

[Nathaniel Smith]

On Mon, Sep 18, 2017 at 9:50 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 18 Sep 2017 09:42:45 -0700
> Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Obviously it's nice when the refcount system is able to implicitly clean
>> things up in a prompt and deterministic way, but there are already tools to
>> handle the cases where it doesn't (ResourceWarning, context managers, ...),
>> and the more we encourage people to implicitly rely on refcounting, [...]
>
> The thing is, we don't need to encourage them.  Having objects disposed
> of when the last visible reference vanishes is a pretty common
> expectation people have when using CPython.
>
>> Why are reference cycles a problem that needs solving?
>
> Because sometimes they are holding up costly resources in memory when
> people don't expect them to.  Such as large Numpy arrays :-)

Do we have any reason to believe that this is actually happening on a
regular basis though?

If it is then it might make sense to look at the cycle collection
heuristics; IIRC they're based on a fairly naive count of how many
allocations have been made, without regard to their size.

> And, no, there are no obvious ways to fix for users.  gc.collect()
> is much too costly to be invoked on a regular basis.
>
>> Because if so then that seems
>> like a bug in the warnings mechanism; there's no harm in a dead Thread
>> hanging around until collected, and Victor may have wasted a day debugging
>> an issue that wasn't a problem in the first place...
>
> Yes, I think Victor is getting a bit overboard with the so-called
> "dangling thread" issue.  But the underlying issue (that heavyweight
> resources can be inadvertently held up in memory up just because some
> unrelated exception was caught and silenced along the way) is a real
> one.

Simply catching and silencing exceptions doesn't create any loops -- if you do

try:
    raise ValueError
except ValueError as exc:
    raise exc

then there's no loop, because the 'exc' local gets cleared as soon as
you exit the except: block. The issue that Victor ran into with
socket.create_connection is a special case where that function saves
off the caught exception to use later.

If someone wanted to replace socket.create_connection's 'raise err' with

try:
    raise err
finally:
    del err

then I guess that would be pretty harmless...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org