capi-sig September 2018

capi-sig@python.org

19 participants
24 discussions

[capi-sig]Re: No C API if it's not doable in Python
by Jeroen Demeyer 07 Sep '18

07 Sep '18

On 2018-09-07 15:17, Victor Stinner wrote: > I'm not sure if we can hide some functions for > regular C extensions, but only give access to Cython? Which problem would that solve? If you're happy with Cython using "internal" functions, why shouldn't other C extensions use those same internal functions?

1 0

[capi-sig]No C API if it's not doable in Python
by Victor Stinner 07 Sep '18

07 Sep '18

Le ven. 7 sept. 2018 à 12:45, M.-A. Lemburg <mal(a)egenix.com> a écrit : > > On 07.09.2018 11:46, Victor Stinner wrote: > > I'm trying to remove functions from the C API which allow to do things > > which are not possible at the Python level: > > https://pythoncapi.readthedocs.io/bad_api.html#no-public-c-functions-if-it-… > > > > It's a request coming from PyPy developers. > > There are always things which you have to be able to do at the > C level but cannot do at the Python level and that's intentional, > since you're working at the C level and want to have direct > access to the data, avoiding copying things around all the > time and creating intermediate temporary objects or buffers for > this. Your motivation here is basically performance, am I right? From what I understood, such optimization in the short term becomes a high technical debt in the long term for PyPy, Gilectomy, and any other Python implementation. Inside CPython, the core and builtin modules, micro-optimizations must be used: abuse borrowed references, macros, access directly to all fields of C structure, etc. But here I'm talking about the public C API used by third party extensions. ... Sadly, I didn't work on this topic yet, so I would prefer to not go further in the discussion, since I have no concrete examples of issue :-) > You also want callbacks to work from both worlds (Python into > C and C into Python), or embed the interpreter, or provide new > ways of working with the existing objects, or work with > code in a way which bypasses the exception machinery (in C > you very often raise exceptions and catch them without the > exception object itself every being created). You should really replace "Python" (the language) with "CPython" in your paragraph. I would like to move away from the current CPython. Victor

3 3

[capi-sig]Re: Open questions about borrowed references
by Jeroen Demeyer 05 Sep '18

05 Sep '18

On 2018-09-05 15:19, Hugh Fisher wrote: > And this happens to be exactly what Apple did when they added tagged > pointers to Objective-C. They told programmers not to access objectptr->isa > directly, instead to use OBJECT_GETCLASS(objectptr). Then later they > started implementing some objects as tagged pointers, and only code > that hadn't been updated broke. OBJECT_GETCLASS was the "stable > ABI" This is a stable API, but I would guess not a stable ABI. Victor Stinner wants a stable ABI.

2 1

[capi-sig]Experiment Python without Py_TYPE()
by Victor Stinner 05 Sep '18

05 Sep '18

Hi, In my previous "Open questions about borrowed references" thread, I asked if it's an issue or not that Py_TYPE() returns a borrowed reference. It seems like most people say that it's not an issue. If you want to discuss if Py_TYPE() is an issue, please contribute to the other thread. Here I only want to share my results. Honestly, I'm still not 100% convinced that Py_TYPE() is an issue :-) I experimented anyway a C API without Py_TYPE(). I chose to add 3 functions/features: * ``Py_GetType()``: similar to ``Py_TYPE()`` but returns a strong reference * ``Py_TYPE_IS(ob, type)``: equivalent to ``Py_TYPE(ob) == type`` * ``%T`` format for ``PyUnicode_FromFormat()`` Most of the usage of Py_TYPE() are the following 5 patterns. For the last one, I'm not sure that the "N" format of Py_BuildValue() should stay since it's also based on borrowed reference. Replacing Py_TYPE() requires a lot of changes, and I made many copy/paste mistakes. If we chose to remove Py_TYPE(), we really need a tool to automate refactoring. Otherwise, the risk of introducing a regression is just too high and send the wrong signals to users. Dealloc:: Py_TYPE(self)->tp_free((PyObject *)self); becomes:: PyTypeObject *type = Py_GetType(self); type->tp_free((PyObject *)self); Py_DECREF(type); Size:: res = _PyObject_SIZE(Py_TYPE(self)) + self->allocated * self->ob_descr->itemsize; becomes:: PyTypeObject *type = Py_GetType(self); res = _PyObject_SIZE(type) + self->allocated * self->ob_descr->itemsize; Py_DECREF(type); Error:: PyErr_Format(PyExc_TypeError, "first argument must be a type object, not %.200s", Py_TYPE(arraytype)->tp_name); becomes:: PyErr_Format(PyExc_TypeError, "first argument must be a type object, not %T", arraytype); Py_BuildValue:: result = Py_BuildValue( "N(CO)O", Py_GetType(self), typecode, list, dict); becomes:: result = Py_BuildValue( "N(CO)O", Py_GetType(self), typecode, list, dict); Victor

1 0

[capi-sig]Re: Open questions about borrowed references
by Jeroen Demeyer 05 Sep '18

05 Sep '18

On 2018-09-05 13:49, Petr Viktorin wrote: > I'm not at a computer right now, but AFAIK if <class int>'s refcount > drops to zero, we still try to deallocate it, and that causes an > assertion and terminates the interpreter. You're right but that's besides the original point. If code uses refcounting correctly (regardless of using an "old" or "new" API), then the class int will never be deallocated.

4 3

[capi-sig]Re: Open questions about borrowed references
by Jeroen Demeyer 05 Sep '18

05 Sep '18

On 2018-09-05 15:20, Victor Stinner wrote: > PyObject *obj = create_object(); > PyTypeObject *type = Py_TYPE(obj); > Py_DECREF(obj); > /* ... */ > /* type is now a dangling pointer */ > > Do you consider that this example uses reference counting "correctly"? No because Py_TYPE() is known to return a borrowed reference. So, if you want to do anything with it besides immediately using it, you need to Py_INCREF() it.

1 0

[capi-sig]Re: static allocation of extension types
by Jeroen Demeyer 05 Sep '18

05 Sep '18

On 2018-09-05 14:51, Hugh Fisher wrote: > As of Python 3.6 a static type cannot be a subclass of a dynamic > type This is true, but I don't really know why. This was added by Serhiy Storchaka in https://bugs.python.org/issue22079 but without much rationale. I'm guessing that heap types are too dynamic to reliably subclass from, but I don't really know... If you really have good reasons for having an extension type subclassing a heap type, I would love to know them. Jeroen.

2 1

[capi-sig]Re: Open questions about borrowed references
by Jeroen Demeyer 05 Sep '18

05 Sep '18

On 2018-09-05 13:32, Petr Viktorin wrote: > Only the type object, <class int>, would be immortal. This is already the case. Extension types are statically allocated, so they are effectively immortal.

3 2

[capi-sig]Re: Open questions about borrowed references
by Jeroen Demeyer 05 Sep '18

05 Sep '18

On 2018-09-05 00:12, Victor Stinner wrote: > Examples of Py_TYPE() usage. My issue is that replacing Py_TYPE() with > a new function which increment the reference counter of the type would > make the code less straightforward. You misunderstood my question. Why would you want to replace Py_TYPE() with a new function which increments the reference counter of the type? It seems to me that you are proposing a solution in search of a problem...

1 0

[capi-sig]Re: Open questions about borrowed references
by Jeroen Demeyer 05 Sep '18

05 Sep '18

On 2018-09-04 23:28, Victor Stinner wrote: > Hi, > > Ah! I succeeded to get a dangling pointer to a deallocated type object > using Py_TYPE(). The example is designed to trigger the bug, but it > proves that it's important to hold a strong reference to the type when > using Py_TYPE(). Maybe it's not an issue for the normal case in the > wild, but at least my example provides that there is an issue in a > corner case which waits to bite you. What's your point really? It's known that Py_TYPE() returns a borrowed reference, so your example code is buggy. It shouldn't surprise anyone that borrowed references can crash CPython.

3 3

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

capi-sig September 2018