[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Neil Schemenauer]

On 2019-04-27, Nathaniel Smith wrote:
> For Py_TRACE_REFS specifically, IIUC the only goal is to be able to produce
> a list of all live objects on demand. If that's the goal, then static type
> objects aren't a huge deal. You can't add extra data into the type objects
> themselves, but since there's a fixed set of them and they're immortal, you
> can just build a static list of all of them in PyType_Ready.

As far as I understand, we have a similar problem already for
gc.get_objects() because those static type objects don't have a
PyGC_Head.  My 2-cent proposal for fixing things in the long term
would be to introduce a function like PyType_Ready that returns a
pointer to the new type.  The argument to it would be what is the
current static type structure.  The function would copy things from
the static type structure into a newly allocated type structure.

We have a kind of solution already with PyType_FromSpec, etc.
However, I think it is harder to convert existing extension module
source code to use that API.  We want to make it very easy for
people to fix source code.

If we can remove static types, that would allow us to kill off
Py_TYPE(o)->tp_is_gc(o).  I understand why that exists but I think
it is quite an ugly detail of the current GC implementation.  I
wonder about the performance impact of it given current memory
latencies.  When we do a full GC run, we call PyObject_IS_GC() on
many objects.  I fear having to lookup and call tp_is_gc could be
quite expensive.

I've been playing with the idea of using memory bitmaps rather then
the PyGC_Head.  That idea seems to depend on removing static type
objects.  Initially I was thinking of it as reducing the memory
overhead for GC types.  Now I think the memory overhead doesn't
matter too much but perhaps the bitmaps would be much faster due to
memory latency.  There is an interesting Youtube video that compares
vector traversals vs linked list traversals in C++.  Linked lists on
modern machines are really terrible.

Regards,

  Neil