capi-sig

capi-sig@python.org

390 discussions

Update on C API changes to hide implementation details

by Victor Stinner

Hi, I pushed many changes in Python 3.9 to hide implementation details: * Add getter and setter functions for PyFrameObject, PyThreadState and PyInterpreterState structures. Examples: PyFrame_GetCode() (frame->f_code) and PyThreadState_GetInterpreter() (tstate->interp); PyFrame_GetCode() returns a strong reference (not a borrowed reference). * Move private functions to the internal C API. * etc. See https://docs.python.org/dev/whatsnew/3.9.html#c-api-changes for the full list. My overall project is described in this draft PEP: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst I created multiple issues to prepare the C API to make more structures opaque: * PyGC_Head: https://bugs.python.org/issue40241 * PyObject: https://bugs.python.org/issue39573 * PyTypeObject: https://bugs.python.org/issue40170 * PyThreadState: https://bugs.python.org/issue39573 * PyInterpreterState: https://bugs.python.org/issue35886 (done in Python 3.8) Related to that, Dong-hee Na and me pushed changes to prevent using Py_TYPE(), Py_REFCNT() and Py_SIZE() as l-value. "Py_TYPE(obj) = new_type" must now be written "Py_SET_TYPE(obj, new_type);". I updated Cython for that: https://github.com/cython/cython/commit/63ab284df20cfe99638f3f352080648c8a3… Many changes are somehow backward incompatible. Some are more incompatible than others. It's not easy for me to estimate how many C extensions will be broken by these incompatible changes. So far, it was quite easy to update C extensions to Python 3.9.0 beta1. One project call directly PyImport_Cleanup() which was described as an "internal" function. This function was moved to the internal C API under _PyImport_Cleanup(). You should not call it directly! In case of big troubles, there is always the ability to revert some incompatible changes, to fix broken projects, before re-applying these changes. I know that these incompatible changes are annoying, but I consider that we have to go through this painful road to enhance (optimize) CPython, but also other Python implementations like PyPy, IronPython or GraalPython. The HPy project proposes a whole new C API written correctly from the beginning. It is a different approach, but it doesn't solve the problem of the big number of existing C extensions which will take several months or years to be ported to HPy. So far, HPy doesn't offer any automated conversion from the current C API to the new HPy C API. But my goal is that CPython C API and HPy C API will converge somehow in the long term. Each small change that I'm making is moving the current C API a little bit closer to HPy! By the way, use Cython for *new* C extension, don't use the C API directly! It's already said in the CPython C API documentation! These C API changes are targeting existing C extensions written with the C API ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.

6 days, 9 hours

Should getter function return a strong reference?

by Victor Stinner

Hi, I added PyThreadState_GetFrame() and PyFrame_GetCode() functions to Python 3.9 (as part of my work to make structures opaque). These functions are basically getter to access PyThreadState.frame and PyFrameObject.f_code members. Example: struct _frame* PyThreadState_GetFrame(PyThreadState *tstate) { assert(tstate != NULL); return tstate->frame; } These functions currently return *borrowed* references. I understood that borrowed references makes it harder to write an efficient implementation of Python: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst#b… For example, PyPy has to convert all items of a list to PyObject when a C extension module calls PyList_GetItem(). Question: should I modify PyThreadState_GetFrame() and PyFrame_GetCode() functions to return a *strong* reference instead? Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 month

Re: [Python-Dev] Problem with instantiating a C extension class from another class

by Musbur

On Sat, 11 Apr 2020 01:07:24 +0000 (UTC) Daniel Scott <scottdaniel760(a)yahoo.com> wrote: > Are you calling PyType_Ready on your types? > > It is difficult to debug this without a full example. If you have a > repo with the code on github or gitlab or bitbucket, feel free to > ping me @mattip to take a look. Hello Daniel, here's a fully self-contained example. Thanks for taking a look! https://github.com/musbur/testmodule

1 month, 3 weeks

PEP: Modify the C API to hide implementation details

by Victor Stinner

Hi, I started a discussion on python-dev about my new PEP "Modify the C API to hide implementation details": https://mail.python.org/archives/list/python-dev@python.org/thread/HKM774XK… The PEP: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst#d… I advice you to participate on python-dev. If you prefer to not subscribe to python-dev, you may reply here. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

1 month, 3 weeks

missing `UCHAR` in `ctypes.wintypes`.

by Kingsley McDonald

hello :) (terribly sorry, i might have accidentally sent an unfinished email before this!) i'm creating a small module for interacting with SCSI devices, and i'm making use of `ctypes` to interact with the win32 API for my windows implementation. in doing so, i've had to define my own version of the SCSI_PASS_THROUGH_DIRECT<https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntddscsi/ns-n…> struct using `ctypes.Structure`. when reading around earlier today, i found the `ctypes.wintypes` module and was happy to see that it could help me write my code to be more consistent with the definitions found in the win32 API. however, the `UCHAR` type appears many times in that struct i mentioned, and yet is not defined in the `ctypes.wintypes` module (in fact, it is commented out in the source with no reason given). i understand that there is probably a reason that i'm unaware of for not having a definition of `c_uchar` in the `ctypes` module. however, i don't understand why an equivalent type (such as `ctypes.c_ubyte`) hasn't been used to define `UCHAR` in the `ctypes.wintypes` module to maintain consistency with the windows API. i know that it is a bit silly to make such a big deal over an alias, but the point of these aliases is to make code look more consistent, and that's hardly the case when there's an exception to the rule! :D if this missing definition was intentional, then what is the reason behind that? or if it has been simply overlooked, then can the definition be created for the sake of consistency? i would love to make the relevant amendment myself if that is the case ^^ thanks! (and sorry again if the other email sent!) kingsley mcdonald :)

1 month, 3 weeks

Macros defined in CPython headers -> move to function-like macros

by Sylvain Corlay

Hi All, I hope I am not breaking any rules by posting here. I maintain a number of libraries and C++ Python extensions and one slightly annoying thing about the Python.h headers is that it includes headers defining macros such as *snprintf*. In C++, this collides with std::snprintf defined in the <cstdio>. One backward-compatible way to make these collision less likely would be to make this macro a function-style macro - so that we could prevent the macro expansion in client code by simply adding parentheses to the calls to the actual functions. (std::snprintf)(foobar) In my opinion, it would make the world a better place for C++ Python extension authors :) Best, Sylvain

1 month, 3 weeks

missing `UCHAR` in `ctypes.wintypes`?

by Kingsley McDonald

hello :) i'm writing a small library

1 month, 3 weeks

Re: [Python-Dev] Problem with instantiating a C extension class from another class

by Nick Coghlan

Hi Musbur, While python-dev is specifically for core development, the "specific interest group" mailing lists are for both change proposals and existing usage questions. I've cc'ed capi-sig on this reply. Cheers, Nick. On Sat., 4 Apr. 2020, 2:02 am Musbur, <musbur(a)posteo.org> wrote: > Hello, > > (I've asked this question before on python-list but only got scarce > and ultimately unhelpful answers. Although this isn't about Python > development itself, I'm assuming that there is more CPython > knowledge on this list than on the other, so please bear with me.) > > I've written a C extension module, each defining a class > (called Series and CLHist, respectively). I can import both modules in > Python and use the classes. Everything works fine, also according to > valgrind. > > Now I want to return a Series instance directly from a CLHist > method, and that immediately crashes Python with a segfault. > > Here's a bit of code and just the _new, _init, and _finalize methods > which I've sprinkled with debugging code. When I instantiate a Series > object from Python, I see output from all three methods. When I call > pyseries_new() from C, none of the three functions are called and it > crashes. > > At some point I thought maybe I should call PyType_Ready() from the C > side before instantiating Series. In a "standard" (system) Python > installation, it printed the desired debugging output and crashed > later. Under a debugging Python installation I built it outputs > nothing except an interesting post-mortem "Fatal Python error: UNREF > invalid object" (see below). > > Here's my C code with the boilerplate stuff left out. > > typedef struct { > PyObject_HEAD > struct clh_series *series; > } Series; > > static PyObject *Series_new(PyTypeObject *type, > PyObject *args, PyObject *kw) { > Series *self; > > self = (Series *) type->tp_alloc(type, 0); > fprintf(stderr, "Series_new(%p)\n", self); > self->series = NULL; > return (PyObject*)self; > } > > static int Series_init(Series *self, PyObject *args, PyObject *kw) { > fprintf(stderr, "Series_init(%p)\n", self); > self->series = NULL; > return 0; > } > > static void Series_finalize(PyObject *self) { > fprintf(stderr, "Series_finalize(%p)\n", self); > clh_series_free(((Series*)self)->series); > } > > /* To create a new Series object directly from C */ > PyObject *pyseries_new(struct clh_series *series) { > Series *pyseries; > > pyseries = PyObject_New(Series, &series_type); > PyObject_Init((PyObject *)pyseries, &series_type); > pyseries->series = series; > return (PyObject *) pyseries; > } > > Here's the debugging output: > > * ob > object : <refcnt 0 at 0x7f1a4e2f97a8> > type : tuple > refcount: 0 > address : 0x7f1a4e2f97a8 > * op->_ob_prev->_ob_next > object : <_Series object at 0x7f1a4e2ee9d0> > type : _Series > refcount: 1 > address : 0x7f1a4e2ee9d0 > * op->_ob_next->_ob_prev > object : <refcnt 0 at 0x7f1a4e2f97a8> > type : tuple > refcount: 0 > address : 0x7f1a4e2f97a8 > Fatal Python error: UNREF invalid object > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/ILMRMQ6… > Code of Conduct: http://python.org/psf/codeofconduct/ >

1 month, 3 weeks

Trying to embed Py3.8 w frozen modules -- correct place to seek help?

by Katie Lucas

Hello. I'm trying to embed a Py3.8 interpreter into a C++ application to use as the "glue" for all the other components. We've come across some complexities -- revolving around the use of frozen modules and multiple interpreter states -- and I'm wondering if this is the correct list to seek some assistance? We can get a single interpreter running. We can add our stream objects and capture the output, we can load our modules (giving the API access) into it and generally everything seems fairly happy in single interp mode. When we try and run a Py_NewInterpreter() to get us more interps to get encapsulation between multiple scripts, it fails to launch (fails to get filesystem's codec). The problem seems to be that the frozen modules added to the launched interpreter are not cached after use. So the cloned interpreter fails to start (it wants either an encoder search path which is null for no filesystem OR to have the encodings module available). The modules don't seem to be being cached because when the import runs them with PyEval_EvalCode, they return nothing. I suspect this is something I've missed about how to configure the original interpreter, but documentation & use-examples for the new split-phase initialisation are rather hard to find. Cheers for any help you can provide, Katie

2 months, 3 weeks

Update on the limited C API

by Victor Stinner

Hi, I pushed a few changes to hide even more implementation details from the limited C API of CPython. See "Changes in the limited C API" documentation at: https://docs.python.org/dev/whatsnew/3.9.html#build-and-c-api-changes I'm interesting to get your feedback on such changes. I know that the PyQt project uses the limited C API. Apart from that, I don't know other C extension using it. One idea would be to consume it directly in CPython for a bunch of C extensions. It would help to check if the API is complete enough for a non-trivial extension. It should also help to detect issues earlier. Currently, CPython only uses the limited C API the "xxlimited" which is basically a test module only written to show how this API can be used. By the way, I just opened "Make PyObject an opaque structure in the limited C API" issue: https://bugs.python.org/issue39573 It's a follow-up on Neil Schemenauer's proof-of-concept of tagged pointer in CPython that he wrote in September 2018. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

3 months

Jump to page:

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

capi-sig