[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Neil Schemenauer]

On 2019-04-24, Victor Stinner wrote:
> The current blocker issue is that the Py_DEBUG define imply the
> Py_TRACE_REFS define

I think your change to make Py_TRACE_REFS as separate configure flag
is fine.  I've used the trace fields to debug occasionally but I
don't use it often enough to need it enabled by Py_DEBUG.

> Being able to switch between Python in release mode and Python in
> debug mode is a first step. My long term plan would be to better
> separate "Python" from its "runtime".

Regarding the Py_TRACE_REFS fields, I think we can't do them without
breaking the ABI because of the following.  For GC objects, they are
always allocated by _PyObject_GC_New/_PyObject_GC_NewVar.  So, we
can allocate the extra space needed for the GC linked list.  For
non-GC objects, that's not the case.  Extensions can allocate using
malloc() directly or their own allocator and then pass that memory
to be initialized as a PyObject.

I think that's a poor design and I think we should try to make slow
progress in fixing it.  I think non-GC objects should also get
allocated by a Python API.  In that case, the Py_TRACE_REFS
functionality could be implemented in a way that doesn't break the
ABI.  It also makes the CPython API more friendly for alternative
Python runtimes like PyPy, etc.

Note that this change would not prevent an extension from allocating
memory with it's own allocator.  It just means that memory can't
hold a PyObject.  The extension PyObject would need to have a
pointer that points to this externally allocated memory.

I can imagine there could be some situations when people really
want a PyObject to reside in a certain memory location.  E.g. maybe
you have some kind of special shared memory area.  In that case, I
think we could have specialized APIs to create PyObjects using a
specialized allocator.  Those APIs would not be supported by
some runtimes (e.g. tracing/moving GC for PyObjects) and the APIs
would not be used by most extensions.

Regards,

  Neil