[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Armin Rigo]

Hi Neil,

On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer <nas-python at arctrix.com> wrote:
> Regarding the Py_TRACE_REFS fields, I think we can't do them without
> breaking the ABI because of the following.  For GC objects, they are
> always allocated by _PyObject_GC_New/_PyObject_GC_NewVar.  So, we
> can allocate the extra space needed for the GC linked list.  For
> non-GC objects, that's not the case.  Extensions can allocate using
> malloc() directly or their own allocator and then pass that memory
> to be initialized as a PyObject.
>
> I think that's a poor design and I think we should try to make slow
> progress in fixing it.

Such progress needs to start with the global static PyTypeObjects that
all extensions define.  This is going to be impossible to fix without
requiring a big fix in of *all* of them.  (Unless of course you mean
to still allow them, but then Py_TRACE_REF can't be implemented in a
way that doesn't break the ABI.)


A bientôt,

Armin.

On Wed, 24 Apr 2019 at 21:17, Neil Schemenauer <nas-python at arctrix.com> wrote:
>
> On 2019-04-24, Victor Stinner wrote:
> > The current blocker issue is that the Py_DEBUG define imply the
> > Py_TRACE_REFS define
>
> I think your change to make Py_TRACE_REFS as separate configure flag
> is fine.  I've used the trace fields to debug occasionally but I
> don't use it often enough to need it enabled by Py_DEBUG.
>
> > Being able to switch between Python in release mode and Python in
> > debug mode is a first step. My long term plan would be to better
> > separate "Python" from its "runtime".
>
> Regarding the Py_TRACE_REFS fields, I think we can't do them without
> breaking the ABI because of the following.  For GC objects, they are
> always allocated by _PyObject_GC_New/_PyObject_GC_NewVar.  So, we
> can allocate the extra space needed for the GC linked list.  For
> non-GC objects, that's not the case.  Extensions can allocate using
> malloc() directly or their own allocator and then pass that memory
> to be initialized as a PyObject.
>
> I think that's a poor design and I think we should try to make slow
> progress in fixing it.  I think non-GC objects should also get
> allocated by a Python API.  In that case, the Py_TRACE_REFS
> functionality could be implemented in a way that doesn't break the
> ABI.  It also makes the CPython API more friendly for alternative
> Python runtimes like PyPy, etc.
>
> Note that this change would not prevent an extension from allocating
> memory with it's own allocator.  It just means that memory can't
> hold a PyObject.  The extension PyObject would need to have a
> pointer that points to this externally allocated memory.
>
> I can imagine there could be some situations when people really
> want a PyObject to reside in a certain memory location.  E.g. maybe
> you have some kind of special shared memory area.  In that case, I
> think we could have specialized APIs to create PyObjects using a
> specialized allocator.  Those APIs would not be supported by
> some runtimes (e.g. tracing/moving GC for PyObjects) and the APIs
> would not be used by most extensions.
>
> Regards,
>
>   Neil
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/armin.rigo%40gmail.com