[New-bugs-announce] [issue41133] Insufficient description of cyclic garbage collector for C API

[Daniel Barkalow]

New submission from Daniel Barkalow <barkalow at iabervon.org>:

Nothing in the documentation gives you enough information to find what you're doing wrong if you have the following bug for an object tracked by the GC: (a) you store a borrowed reference to an object whose lifetime is always longer that the buggy object (and never decrement it); (b) you visit that object in your tp_traverse.

Despite the bug, everything works nearly all of the time. However, when it fails, the effect makes no sense from a documentation-only understanding of the garbage collection (unless Python is built with --with-assertions, perhaps). The only sequence in which it fails is:

* All of the objects that will reference the victim get moved into an older generation.
* The victim gets exactly as many buggy references from objects in its own generation as there are good references, and no good references from objects in its own generation.
* Garbage collection runs for only the younger generation.

At this point, the victim is marked as garbage despite the fact that there was a good object referencing it at all times.

Reading the Python source, it becomes clear that the garbage collector handles references from older generations not by traversing the objects in older generations, but by comparing the count of references from other young objects to the usual reference count. That is, visiting an object you don't hold a reference to may cause that object to get collected, rather than protecting it, even if there are other reasons not to collect it.

The best fix would probably be a warning in the documentation for tp_traverse that visiting an object you do not hold a strong reference to can cause inexplicable effects, because the intuitive guess would be that it could only cause memory leaks.

It would probably also be worth mentioning in the documentation of the garbage collector something about its algorithm, so people have a hint that references from older objects aren't necessarily sufficient to overcome buggy use of the C API.

----------
assignee: docs at python
components: C API, Documentation
messages: 372445
nosy: docs at python, iabervon
priority: normal
severity: normal
status: open
title: Insufficient description of cyclic garbage collector for C API
versions: Python 3.7, Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue41133>
_______________________________________